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ABSTRACT 

This project examines anaphora (the linguistic device 
of abbreviated subsequent reference to a concept) in information 
retrieval (IR) systems in order to develop procedures to recognize 
anaphors in text and distinguish between anaphoric and non-anaphoric 
uses of a given term, estimate the number of anaphors appearing in 
bibliographic records, and assess the effect on retrieval performance 
when anaphors are replaced by their referents. In the first phase of 
the study, rules were developed to form the basis for an automatic 
procedure to recognize anaphoric terms in bibliographic databases. An 
examination of the titles and abstracts of 600 documents revealed 
that only 3.67 true anaphors occurred in the average abstract, 
suggesting that the effect of treating these terms in some way to 
improve retrieval performance might be slight. In the second phase, 
12 term weighting schemes were used to determine the relevance of 
each document to the corresponding query, and user's relevance 
judgements for the same searches were compared with the system's 
judgements for (1) searches using abstracts in which anaphors had 
been replaced with their referents, and (2) searches using abstracts 
with unresolved anaphors. These comparisons yielded mixed results, 
indicating that a straightforward substitution of referents for their 
anaphors will not improve retrieval performance in the majority of 
cases. It is concluded that future studies which treat document 
length more explicitly and study documents on an individual level are 
necessary. A bibliography is provided, and five lengthy appendices 
include the preliminary test and functional indexes, the retrieval 
experiment and functional indexes, results of the linguistic 
analysis, test results of rule sets, retrieval test results, and 
summaries of statistical results for searches of INSPEC and PsyclNFO. 
(KM) 
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ABSTRACT 



Anapho rat i m thm 1 i ngu i « t i c dev i c& of abb rmv i ti tad 
subMquant rafaranca to a concapt. This raaaarch pro j act 
was basad on tha hypothaaia that within docuaant fraquancy 
(VOF) of a tarst and ultiaataly ratrieval parforaanca of a 
ayataa uaing «0F« uould ba affactad by tha raaolution of 
anaphora (raplacaaant of ita anaphor with ita rafarant) 
within ita taxt of a docuaant. In ordar to teat tha 
hypothaaiat a two-phaao invaatig^tion waa iaplaaantad. 

In tha firat phaaa* all potential anaphora in a randoa 
aaapla of 300 abatracta froa aach of two databaaaa wara 
idantifiad. Each occurranca of anaphora waa than axaainad 
in ordar to dataraina if tha tara actually functioned 
anaphorically. Froa thaee obaarvationa* pattarna aaerged 
which were then developed into rulaa that captured the 
ayateaatic ragularitiea of functional anaphora. Tha rulaa 
were teeted by at leaat three people to dateraine whether 
the rulee accurately dietinguiahed functioning anaphora froa 
potential anaphora. 

I n tha aac ond phaae o f the p r o j ac 1 1 24 qua r i ea » 
abatracta retrieved froa coaputerized eearchea on the 
queriee. and relevance judgaente on aach retrieved docuaant 
were aalected froa a pravioue reeearch project. All 
functioning anaphora within tha abatracta ware raaolved by 
hand. Twelve tera weighting acheaee were uaed on the baaia 
o f deter a i n i ng re 1 evance o f each docuaant to i ta 
correeponding query. Two atatiatical relationahipa ware 
then coaparads 1) between the ueer'a relevance judgaent and 
the ayatea'a judgaent baaed on the unreeolved abatracta* and 
2) between the ueer'e relevance judgaent and the ayatea'a 
Judgaent baeed on the reeolved abatracta. If tha latter 
relation ia atronger than tha foraar* then a foraal 
traataent of anaphora in bibliographic retrieval poaitively 
affeote ayataa parforaanca. 

Reeulte of the coapariaona were aixed. In aoae 
inatancea* the reeolved docuaanta produced a aignif icantly 
better correlation between uaer'a Judgaanta and ayetea^a 
Judgaantat while in other inatancea* tha oppoaite occurred. 
Tha findinga that reeolution of anaphora aay increaae the 
perforaanca of a ratrieval are far froa concluaive. It ia 
clear that future etudiee of anaphora in inforaation 
ratrieval auat be treated in a acre coaplax aannar than waa 
atteaptad here. 
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OVERVIEV 



In frM-t*xt inforution r«tri«val (IR) sytttHBt all 
non-trivial words in thm docuMnt mtm xjmmA to raprasant tha 
contant of that docuaant. In tha daaign of thaaa ayataaat 
it ia raaaonabla to baliava that tha aora oftan a tara ia 
rapoatad* tha aora likaly it ia that tha tara rapraaants a 
aajor concapt of tha doeuaant. It ia for thia raaaon that 
IR ayataaa uaight tha iaportanca of a givan tara as a 
function of ita fraquancy of occurranca within tha docuaant. 
Kowavar* a atraightforuard count of aach word typa doaa not 
go far anough bacaum it axcludas waya in which tha aaaa 
concapt can ba rapraaantad by othar worda. Raaoving 
auffixaa and coabining aynonyaa ara two aathoda that ara 
uaad to aaka tha raaulting tara waighta battar raflact tha 
trua praaanca of a concapt in a docuaant. 

Anothar way in which an inatanca of a concapt can ba 
"hiddan* in a count of tara fraquanciaa ia through anaphoric 
rafarancat whara* for axaapla* a pronoun rapraaanta a aajor 
concapt d i acuaaad a 1 aa whara i n tha docuaant . Though 
anaphora has bean aentionad by aavaral raaaarchara in 
inforaation acianca* vary littla ia known about tha axtont 
of anaphora in bibliographic databaaaa or how an axplicit 
traataant of anaphora aay changa tara waighta and 
conaaquantly ratriaval parforaanca. 

Thia raport docuaanta tha f irat invaatigation of 
anaphora in IR. Ilora apacif ical ly> our objectivaa wara to: 

1. Davalop proceduraa to racogniza anaphora in text and to 
diatinguiah batwaen anaphoric and non-anaphoric uaaa of 
a givan tara. 

2. Eatiaata tha nuabar of anaphora appaaring in 
bibliographic racorda. 

3. Aaaaaa tha affact on ratriaval parformanca whan anaphora 
ara raplacad by thair rafaranta. 



Thaaa objactivaa ara addraaaad in tha naxt two aactiona of 
thia raport. 

Tha firat aaction ia baaad on an axaaination of 
axiating linguiatic thaory conbinad with a datailad atudy of 
a randOB aampla of 600 docuaanta (titlaa and abatracta). 
142 worda wara idantifiad aa potantial anaphorat though 
aaong tha docuaanta atudiad only 95 of thaaa actually wara 
praaant. Thaaa worda wara organizad into tan claaaaa and 
for aach* rulaa wara davalopad to dataraina whathar a givan 
tara f unc t i onad anaphor i ca 1 1 y aa it waa uaad i n tha 
docuaant. Thaaa rulaa can fora tha baais for an autoaatic 



procsdur* to racogniza anaphoric taraa in bibliographic 
databaaaa. An axaaination of tha 600 docuaanta diacovarad 
that only 3.67 trua rnaphora occurrad in tha avaraga 
abatract — auggaating that tha affact of traating thava 
taraa in aoaa uay to iaprova ratriaval parforaanca aiaht ba 
alight. 

Tha aacond aaction of thia raport praaanta tha raaulta 
of our axaaination of tha third objactiva. Thia atudy ia 
tha praaiaa that anaphora ara uaad by authors to 
avoid rapatition and aa aucht thay ara likaly to rapraaant 
tha Bora important concapta in a docuaant. Tharaforat 
raplaeing all anaphora uith thair rafaranta will changa tara 
fraquanciaa in auch a way ao aa to iaprova ratriaval 
parforaanca. A poat-ratriaval axpariaant uaa conductad 
aaking uaa of 12 axiating quariaa for aach of tha two 
bibliographic databaaaa. All docuaanta ratriavad by thaaa 
quariea uara axaainad to idantify all trua anaphora. Than, 
by hand, aach of thaaa anaphora uaa raplacad with ita word 
or phraaa rafarant. Thia procaaa changad tha fraquancy of 
oceurranca of uorda in tha docuaant. and tharafora tha 
pradictad ralavanca of tha ratriavad docuaanta uaa alap 
changad. if tha procaaa of raplaeing anaphora with thair 
rafaranta iaprovaa ratriaval parforaanca. than tha raviaad 
aat of tara fraquanciaa ahould pradict docuaant ralavanca 
battar than tha original fraquanciaa. 

Tha raaulta of tha atudy ara aixad. Traating anaphora 
doaa iaprova ratriaval for aavaral quariaa though all 
elaaaaa of anaphora do not contributa aqually to thia 
iaprovaaant. Thara ara alao inatancaa in uhich ratriaval 
parforaanca dacraaaaa uhan givan elaaaaa of anaphora ara 
raplacad uith thair rafaranta. Houavar. for tha aajority of 
quariaa thara ia no affact of traating anaphora in thia uay. 
Tha aajor concluaion of thia uork ia that a atraiahtforuard 
aubatitution of anaphora for thair rafaranta uill not 
iaprova ratriaval parforaanca in tha aajority of caaaa. Va 
raaain convincad. houavar. that the baaic praaiaa underlying 
thia raaaarch ia trua. viz.. that anaphora ara uaad to 
abbraviata aubaaquant aantiona of tha aora iaportant 
concapta in a docuaant. Tharafora. tha study of anaphora in 
IR raaaarch ahould not be abandoned* rather, other aeana of 
iaolating the reference to key concepts need to be explored. 

Tvo avenuea of additional uork ara propoand. Firat. 
docuaant length needa to be treated acre explicitly. Vhen 
an anaphor ia replaced it often ia not a one-uord for 
one-' rrd eubatitution.' Inatead. entire phraaae aay be edded 
to trie docuaant. increaaing tha nuaber of trivial taraa acre 
than the number of inatancae of key teraa. Becauae ranking 
foraulas tend to be aenaitiva to the total nuaber of words 
in a document, retrieval parforaanca can deteriorate efter 
an anaphor ia replaced. Another approach to liaiting tha 
inoreaae in docuaant length ia to edit the aubatitution 
proceee by allouing only taraa that appeared in the query to 
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hm «d(i«d to thm dloeuB«nt wh«n an anaphor is replacad. Th* 
aacond araa of additional work ia to atudy docuaanta on an 
individual laval. By focuaing on ratriaval parforaancat our 
laval ot analyaia had to ba tha quary. Replacing anaphora 
with thair rafaranta aay affact individual documenta quita 
diffarantly and tha ovarall affact on tha quary would ba 
aoaa "avaraga" of what happanad to tha individual docuaanta. 

At thla tiaat thaaa two araaa of futura work aaaa to 
hold tha aoat iaaadiata proaiaa for tapping tha potential of 
uaing tha full aaaantic content of anaphora to iaprove 
inforaation retrieval ef fectiveneaa. Thia potential effect 
exiata not only for docuaant abatracta uaed in free-text 
eearching* but alao in other araaa of inforaation retrieval 
work that uaa naturally occurring texte auch as uaera* 
queriea or full-text docuaanta in a quaation-anawering 
ayatea. 
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A Study of OiMoutM Anaphora 
in Sciantific Abstracts^ 

Eliaabath Liddy* Suaan Bonai* Jaf£ray Katzar tc Elizabath Oddy 

School of Information Studiaa* 

Syracuaa Uni varsity* Syraeuaat Nau York 13244 

Introduction 

Huch of tha work that inforaation ratriaval ia involvad in 
aakaa uaa of naturally occurring taxta such aa uaara* quariaat 
abatracta in a fraa*taxt ratriaval ayataa* or fuU-taxt docuaanta 
in a quaation*anauarin0 ayataa. To davalop auccaaaful ayataaa in 
any of thaaa araaa raquiraa an adaquata handling of tha uhola 
ranga of linguistic phanoaana that axhibit thaaaalvaa in natural- 
ly occurring taxt. Thay aay ba uord*laval (aorphology) or 
aantanca*laval (ayntax) phanoaana or thay aay ba diacouraa laval 
phanoaana which bacoaa a factor uhan analyzing unita of taxt 
largar than a aingla aantanca. Oaaignara of inforaation ratriaval 
ayataaa hava alraady laarnad to apply linguistic knouladga daval- 
opad in both aorphology and syntax* For axaapla* aorphology has 
contributad tha tachniqua of ataaaing which conflataa tarainolo* 
gical variants to thair ataa* whila tha autoaatic idantif ication 
of noun phraaaa for uaa as indaxing phraaaa uaaa ayntaetic analy* 
aia (1]« Howavart inforaation ratriaval ayataaa which aanipulata 
chunka of connactad taxt auat alao attand to tha taxt laval pha- 
noaana which hava aora racantly coaa undar atudy in diacouraa 
linguiatica. Aaong tha linguiatic davicac of concarn at tha dia- 
couraa laval ara anaphora* cataphora* allipaiat aubatitutiont 
parallaliaa and intar-aantantial conjunction. 

1 Baaad on an articla accaptad for publication in JASIS. 
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Tht dlacourM Uval phMoMnon dsalt with In this ptp«r can bm 

■oat inolusivaly rafarrad to aa diaoouraa anaphora. Thia uaa of 

tha tara anaphora raflacta coaaon uaaga in diaoouraa linouiatica 

rathar than that of Choaaky and linguiats of tha tranaforaationai 

graaaar achool who uaa tha tara *anaphor* in a aora narrowly 

dafinad aanaa* Uhila Choaaky ia concarnad uith dataraining tha 

axact conditions undar uhioh pronouna function within ona san-* 

tanca» our concern ia with all anaphorio-^typa rafarancoat uhathar 

within or acroaa aantanoa boundariaa* Diaoouraa anaphora can ba 

dafinad &a abbraviatad subaaquant rafaranca and ia aoat coaaonly 

axaaplifxad byt but not liaitad tot tha uaa of pronouna* Exaa- 

plaa of diaoouraa anaphora can ba aaan in tha following sxcarpt 

tihara tha tara **countaridantif ioation'* ia actually uaad only 

oncat but tha concapt ia aaaantically praaant a total of thraa 

tiaaa oinca it ia anaphorioally rafarrod to twica aorat onca by 

^thia aachaniaa** and onoa by **it** 

Countaridantif ication ia a aaohaniaa that aakaa changaa 
within tha paychie atructura of tha individual. Thia 
aochantaa diffara froa nagativa idantH ication in that 
Ul uaaa tha aggraaaiva anargiaa..** 

Huaana (a.gr indaxarat or uaara judging docuaant ralavanea) aan-* 
tally raaolva anaphoric rafarancaa and appaar abla to taka abbra- 
viatad rafarancM into oonaidaration in conatructing appropriate 
aantal rapraaantationa of taxt. Thia ia facilitated by tha fact 
that in exj^oaitory texte a new entity (a concapt or object) ia 
ueuelly introduced to the reader in ita fuUeett aoat explicated 
fore* A poaaible eyntax for auch a noun phraee iai 

det ♦ adj| ♦ •••• adj^ ♦ noun ♦ prep phraee/ rel elauae 
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Full first-Mntion is uMd in ordsr to firaly Mtabliah a virtual 
inatanca of tha antity in tha aind of tha raadar. Having auccaaa* 
fully anchorad tha laxical raalisation to a aantal rapraaanta- 
tiont furthar coaaanta can ba aada about that antity without 
rapaating all tha pra- and poat-aodif iara uaad in tha firat* 
aantion raalization forat or avan without uaing tha noun itaalf. 

Tha ranga of poaaibla aubaaquant*-aantion raalisation foraa 
rhich uould ba conaidarad anaphoric rafarancaat includas 

0 datarainar ^ mmmm noun 

0 datarainar ^ ganaral noun 

0 pronoun 

All of thaaa aubaaquant-aantion foraa ara ahortar and eonvay laaa 
inforaation than full f irat-aantiona. Houavar» thaaa foraa do 
coaaunicata auccaaafully and unaabiguoualy to a raadar bacauaa 
all tha taxt naad do ia raaind tha raadar which antity ia baing 
aantionad» rathar than craata a naw aantal rapraaantation. 

Although huaana aaldoa ancountar difficulty in racognizing an 
anaphor and corractly idantifying tha rafarant of tha anaphoric 
axpraaaion in text» diacouraa anaphora raaaina ona of tha taxt 
laval phanoaana at ill poaing aubatantial difficultiaa for tha 
aany fialds that ara attaapting to aaka uaa of naturally occur- 
ring taxta. In inforaation scianca* tha nacaaaity for racognizing 
and raaolving anaphoric rafarancaa iapacta on 1) natural languaga 
undaratanding» 2) quaation-«nawaring» 3) autoaatic axtracting» 4) 
quary analyaiat and 5) bibliographic ratriaval. 

Ifatural Unauaaa Ondarp^ffn4^nflt A natural languaga undaratanding 
ayataa naada to build a aaaantio rapraaantation of tha taxt baing 
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procMMd. In order to do this succoMfully* thm difficult task 
is not in sccurstsly rsprsssnting ths sssning of ssoh nsu input 
ssntsncs singly* but rsthsr in sppropristsly cosbining ths sssn* 
ing of sll individual ssntsncss to fors a rsprsssntation of ths 
aggrsgatad aaaning of ths tsxt. It is s sattsr of intsrprsting 
nstf inforaation in light of ths old and of connscting nsu infor* 
sation to ths appropriats old inforaation in ths rsprsssntation* 
so that a cohsrsnt uhols rssults. It is not unsxpsctsd» thsn* 
that ths task of eorrsctly intsrprsting discourss anaphora is 
ssssntial for building intsgratsd rsprsssn tat ions of ssaning for 
natural languags undsrstanding systsss (2 I. 

QusB t i on'-Ansusr 1 no i Qusstion*anstfsring systsss say bs of two 
typss* both of uhich rsquirs handling of discourss anaphora. 0ns 
approach to qusstion**ansasring systsss is to build sssantic rsp^ 
rsssntstions of both ths tsxts in ths systss and ths ussrs* qusr- 
iss and uss ths Isttsr rsprsssntstions to find spproprists 
ansusrs aaong ths forssr. If this is ths spproach taksn* ths 
rationals givsn abovs for anaphora rssolution tschniquss in 
M.L.U. holds for this task as iisll. An altsrnstivs spproach to 
qusst ion**ansusring has bssn sttssptsd by John 0* Connor C 3 ]# 
O'Connor sttssptsd to provids snstisrs to qusriss by rstrisving 
snsusr-providing passagss fros ths sctusl tsxt of ths docussnt 
rsthsr than building an intsrssdists sssantic rsprsssntation of 
ths tsxt. His rssults vsrs vsry prosising* but O'Connor suggsstsd 
thst furthsr isprovsssnt could bs gainsd if it ssrs possibls to 
loosts in tsxt ths fully sxplicstsd sxprsssions uhich ars subss^ 
qusntly rsfsrrsd to in sn sbbrsviatsd ssnnsr by anaphoric cluss 
such as *this«f 'thsss* snd 'thoss^ 
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AutoMtic ExtfMtinot Paica's work C4] on autontic •xtraeting 
clmrly raeoonisEad thm nMd for attanding to anaphoric rafaranca 
in taxt. In ordar to autoaatically ooapila a coaprahanaiblat 
aubatantiva axtractt Paica found it nacaaaary to aatatliah a Hat 
of *clua uorda' (a.g. Mt«* *thaa** *aiailarN *bothM which 
indieatad that if tha particular aantanca in which thaaa uorda 
oecurrad uaa to ba inoludad in tha axtractt it would ba nacaaaary 
to loeata and includa tha aarliar taxt in which thaaa anaphoric 
rafarancaa wara aora fully axplicatad. 

Ouarv Analvaia* Raaaarch currantly undarway by Oddy t51 (aaa 
alao Balkint Oddyt Brooks (61) into an inforaation aaakar's atata 
of knowladga on tha topic or problaa which coapallad thair intar- 
action with tha inforaation ratriaval ayataat includaa tachniquaa 
for analyzing and rapraaanting ralationahipa batuaan concapts in 
tha uaar*a problaa atataaant. Thaaa ralationahipa ara currantly 
coaputad froa quita atiparficialt aainly atatiaticalt cbaractaria* 
tica of tha taxta. Alaot tha taxta ara tranacripta of oral uttar- 
ancaa with copioua uaa of anaphora. Hancat raaolution of dia- 
couraa anaphora would undoubtadly affact tha darivad 
rapraaantation of tha uaar's atata of knowladga* 

gtfrUgHrBPtlt9 BytrtfYtP in fraa-taxt docuaant ratriaval aya- 
taaat tha problaa of corractly raoogniaing and raaolving aubaa- 
quant rafarancaa ia iaportant baoauaa aany of tha atatiatical 
aathoda of dataraining which docuaanta ara to ba ratriavad in 
raaponaa to a quary aaka uaa of fraquancy counta of taraa* For 
thia count to ba a trua aaaaura of caaantic fraquanciaa* it would 
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apptar that tha aaaantically raducad auhaaquant rafarancaa ahould 
ba raaolvad by thair aarliatt aora fully apaeifiad rafaranta in 
taxt. A tachniqua in aany axpariaantal and a faa oparational 
docuaant ratriaval ayataaa ia to aaight tha taraa of a docuaant*a 
fraa-taxt rapraaantation <titla and abatraot) on tha baaia of 
tara fraqiianciaa. Tha inforaation ratriaval ayataat applying a 
aiaiUrity aaaaura batuaan quary and docuaant rapraaantationat 
uill than do a baat^aatch aaarcht ratriaval t and ranking of docu*- 
aanta for tha uaar. Thia tachniqua ia baaad on tuo apparant 
aaauaptiona: 1) that fraquancy of occurranca ia a good indication 
of tha dagraa to uhich a piaca of taxt ia about a cartain tara* 
and 2) that an adaquata aaana of dataraining aaaantic fraquancy 
of a concapt ia by counting all axplicit occurrancaa of a tara. 

Houavarf tha thaory bahind diacouraa anaphora pradicta that an 
adaquata aaaaura of fraquancy of occurranca of a concapt raqui raa 
that all iaplicit occurrancaa of that tara ba takan into account* 
In bibliographic ratriaval t thia aould aaan that tha fraquancy 
count of docuaant taraa aftar raaolution of all anaphora aould 
battar rapraaant ahat tha docuaant ia about and that raaolving 
anaphoric taraa in abatracta would aignif icantly iaprova ratriav* 
al raaulta by obtaining for tha uaar a rankad ordaring of docu- 
aanta aora truly raf lactiva of tha docuaanta' dagraa of ralavanca 
to tha iiaar^a quary* 

Evan though tha araaa of work in inforaation acianca diacuaaad 
abova naad to ba concarnad with diacouraa anaphorat no atudy 
axiata ahich providaa aithar 1) baaa-^lina quantitativa data on 
tha axtant to which thia phanoaanon axiata in a taxt-typa uaad in 
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inforMtion sci«nc«t or 2) insight into uhsthsr thm um of ana** 
phoric rafarancaa in auch a taxt typa ia rula-oovarnad anough to 
parait davalopaant of algorithaa for autoaatically datacting and 
than raaolving anaphoric rafarancaa. In fact* a racantly pub-* 
liahad invaatigation by Fidal C73 of thoaa aapacta of fraa-taxt 
uhich aight iapact on ratriaval* aantionad no concarn about ana- 
phoric rafarancaa. It ia hopad that tha banchaark daacriptiva 
data and faaaibility of autoaatic raccgnition and raaolution of 
anaphora providad by thia atudy aay ba uaaful to thoaa araaa of 
uork on uhich tha praaanca of anaphoric taraa haa an iapact. 

Study 

Our uork conaiatad of datacting occurrancaa of anaphoric raf- 
arancaa and coaputing baaa-lina counta* aa uall aa davaloping 
rulaa uhich uould captura in algorithaic fora tha daeiaiona aada 
by huaan proeaaaora both aa to uhathar a tara ia anaphoric or not 
and to uhat ia ita propar raf arrant. Bafora thaaa taaka could ba 
attaaptad* houavar* aoaa praliainary atapa uara raquirad. 

Tha firat taak uaa to davalop a liat of all thoaa taraa con- 
aidarad potantially anaphoric. Having locatad no auch pra- 
aatabliahadt all-incluaiva liat in tha litaratura* ua cospilad a 
liat froa graaaar hooka* particularly Quirk* Graanbaua* Laach li 
Svartik CSl* linguiatic uorka daaling uith linguiatic davicaa 
adding to tha cohaaion of a taxt* auch aa Halliday It Kaaan [93* 
and Griaaa [101* and prior invaatigationa into aoaa aubaat of tha 
phanoaana of diacouraa anaphora (Vabbar* CUl* Sidnar* (123* and 
Hirat* C23). Thia raaultad in tha aat of 142 potantial anaphora 
(P.A.a). liatad in (Figura 1). 
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'-jiliitliffiiffTiTLiU 



•abovs 

•additioMl 

aforMentiontd 

aforsraid 
•all 

•anothar 

anothar*0 
•any 

anybody 
•anyona 
•anything 
•both 
•did 
•do 
•doas 
•doing 
•donft 
•aach 

aighth 
•aithar 
•alaa 

alaa'a 

•anough 

•aqual 

•avary 

avarybody 

avaryona 

avaryona*a 

avarything 

• fan 
•fawar 

fauaat 
•fifth 
•first 

foraaantionad 
•fcraar 

foraar's 
•fourth 
•ha 
•har 
•hara 

hara 
•?sarMlf 
•hia 
•hiaaalf 
•his 

• I 



•idantical 

•idantically 

•it 

•ita 

•itaalf 

•laat 

•lattar 

lattar*s 
•laaat 
•lass 

likauiaa 
•littla 
•aany 

aa 

aina 
•aora 
•aost 
•auch 

■y 

aysalf 

•naithar 

•ninth 
•no 

nobody 
•nona 
•nothing 
•ona 
•ona*s 

onas 
•onas* 
•othar 

othar's 
•otbara 
•our 

ours 

oursalvas 

•S 

•S's 
•Ss 
•Ss' 
•acaa 
•aacond 
savanth 
•savaral 
•aha 

•aiailarly 

•aixth 

•so 



•soaa 

soaabody 
aoaabody*a 
aoaaona 
Boaaona'a 

•aoaathing 

•auch 
tanth 

•that 

•tha 

•thair 
thairs 

•thaa 

•thaasalvas 

•than 

•thara 
tharaat 
tharafor 
tharafroa 

•tharain 
tharainto 
tharaof 
tharaon 
tharaout 
tharato 
tharaundar 
tharaoith 

•thaaa 

•thay 

•third 

•this 

•thosa 

•thua 

•us 

•vies varaa 
•ua 

•ahara 

•ahich 
•aho 
•ahoa 
•ohoaa 

you 
•your 

yours 

yoursalf 

youraalvas 



A classification sehaaa (Figura 2) aas than iaposad on thasa 
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P.A.B flo that uork could procMd at tha claaa or aub-claaa laval. 
Tha claaa diatinctiona vara aada on a functional baaia guidad by 
tha intuition gainad froa a aaall faaaibility atudy which invas- 
tigatad uhathar racognition and raaolution tachniquaa uould ba 
ganaralizabla at tha functional claaa laval. 



1* Cantral Pronouna 

a. Paraonal Pronouna - hat hiat it 

b. Poasaaaiva Pronouna - hiat hart thair 

c. Raflaxiva Pronouna - itaalft thaaaalvaa 
2. Noainal Daaonatrativaa - thiat tbaaat thoaa 
3* Ralativa Pronouna - aho* uhicht vhara 

4. Noainal Subatitutaa - abovat foraart ona 

5. Pro*varb - do 

6* Indafinita Pronouna - anyt aach» aany 
7. Pro^adjactivaa - anothart idantical 
8* Pro*advarbial0 - aot aucht aiailarly 
9* Subjact Rafarancaa - St Sa 
10. Oafinita Articla - tha 

FlQura 21 Claaoaa of Oiacouraa Anaphora oith axaaplaa 



Ifoat of tha taraa ahich ara capabla of anaphoric rafaranca can 
alao par f era othar funeticna in taxt and mm a raault ahould ba 
conaidarad mm only potantial anaphora (P. A. a). Any ayataa vhlch 
adaquataly handlaa a u baaquant rafaranca in taxt firat naada m 
aaana for dataraining in a particular i^iatanca if a P. A. ia actu- 
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ally a functioning anaphor (F*A«)« Although Moat huaana can 
quita aaaily daeida in a apacific inatanca uhathar a tara ia 
baing uaad anaphorically or nott tha p^aciaa linguiatic avidanca 
on uhich theaa dsciaiona ara aada ia not avai labia in tha litara* 
tura and naada to ba dalinaatadt ao that algorithaa can ba writ* 
tan for accoapliahing tha aaaa taak* Tharaforat ua conductad a 
atudy to aaa uhathar it would ba poaaibia to davalop rulaa uhich 
could ba auccaaafully appliad by indapandant judgaa and raault in 
a claar aaparation batuaan thoaa inatancaa uhara a P.A. ia aiaply 
a P.A* and thoaa inatancaa uhara a P.A. ia an F.A. Succaaa uith 
thaaa rulaa uould auggaat tha faaaibility of davaloping aachina- 
iaplaaantabla algoritbaa to aaka tha aaaa diatinctiona. 

To urita auch algorithaat it ia nacaaaary to look at a corpua 
aufficiantly larga that tha ragularitiaa of ayntax and laxical 
choica uhich uould aarva aa tha baaia of thaaa rula-baaad algor- 
ithaa uill axhibit thaaaalvaa. Unfortunatalyt auch of tha pravi* 

r' 

oua work in linguiatica on anaphora haa uaad contrivad taxta or 
eorpuaas too aaall to ganaraliza froa. Sot for a corpua on uhich 
to urita and taat rulaa for racognizing uhan a P.A. ia an F^Act 
ua drau 600 abatraeta at randoat 300 aaeh froa tuo oparational 
docuaant ratriaval databaaaa^t 1) PayelNFO - uhich containa 
abatraeta of docuaanta raporting on the bahavioral aciancaat and 
2) INSPEC - uhich containa abatraeta of docuaanta raporting on 
anginaaring and coaputar acianca. Thia coabinad aat contained 
occurrancaa of 95 P. A. a (atarrad taraa in Figure 1) froa tha pre* 
liainary coapilation of 142 P. A. a* Tbeee 95 threat on uhich the 
foUouing uork ie baaadt ware aaaigned to one of the 10 claai 
of anaphoric taraa (aae Figure 2). 
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Ihm bavic proctdur* which um foUoutd in dtvaloping and tMt- 
ing thtt P.A.-to*F.A. rulM is outlined mm folloust 
1* For MCh cla0v» all abatracta containing occurrancaa of 
taraa of that claaa vara collactad. Tha axact nuabar of 
abstracta drawn for aach claaa corralatad roughly uith tha 
fraquancy with which taraa of that claaa occurrad. 
2. For aach occurranca of a P.A.t an intallactual daciaion waa 
aada aa to whathar tha P*A* waa an F«A*« Thia providad tha 
baaic auaoary data baing raportad hara* 
3* Vhila doing tha abova atept pattarna bagan to aaarga froa 
tha taxtas tha pradictability of contextual inforaation in 
dataraining whathar tha uaa of tha tara waa anaphoric or 
nonanaphoric bacaaa avidant* 

4. Froa thaaa obaarvationst» P.A**to-F«A* rulaa wara writtan 
which captura tha ayataaatic ragularitiaa which* whan 
ancoded in algorithaat willt wa hopat raplaca huaan intui*' 
tiva decision aaking. Thaaa ragularitiaa ara aithar in tha 
lexical environaent in which anaphoric/nonanaphoric uaa of 
a tare can be predicted to occurt or the particular syntac- 
tic conatruction indicating anaphoric /nonanaphoric uaa*' 

5. The P*A*-to-*F*A* rule seta for aach classt sub*claaa* or 
tara were elightly reworded where necessary using a lees 
linguistically oriented vocabulary. Each rule set was giv* 
en to at least three Judgee who applied thes to a aubeet of 
the original 600 abetracta. Each rul» was teeted on ten 

« Rules have not aa yet been developed for claaa lOt the definite 
articlSf due to the unpredictability of the contexts in which 
*the* eppeers. Following enalysis of the reeults of the 
retrieval expsriaantt rules will be atteapted if the reeults 
warrant algoritha developeent for this class. 

P.2 
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diffsrent occurrsncM of thm t«»(s) to which tha rula 
applied. If th«r« umrm Imwm than 10 occcurrancas* all of 
tha aval labia occurrancaa vara taatad* 

Raaulta 

Tha raaulta raportad hara ara of a tuofold natural 1) auaaary 
data on diatribution of P.A.a and F.A.a in abatracta; and 2) muc^ 
emmu of uriting rulaa for uaa in dataraining uhathar a P.A. ia an 
F.A. 

Diatributional Analvaia 

Tha auaaary data indicataa that tha linguiatic phanoaanon of 
diacouraa anaphora axhibita itaalf to a graatar axtant in PaycIN** 
FO than in INSPEC. 

Tabla 1 ahoua tha aaan occurranca of P.A.a par abatract to ba 
13.2 for tha PaycINFO abatracta* and 10.08 for tha INSPEC 
abatracta* with a aaan occurranca of 11.64 P. A. ft par abatract 
acroaa tha coaplata aaapla of 600 abatracta. Tha aaan occurranca 
of F.A.a par abatract ia 4.49 for tha PaycINFO abatracta* and 
2.66 for tha INSPEC abatracta* with a aaan occurranca of 3.67 
F.A.a par abatract acroaa thm coaplata aaapla of 600 abatracta. 
Thaaa praliainary raaulta auggaat that tha phanoaanon of dia- 
couraa anaphora haa a graatar iapact on a natural languaga taxt- 
handling ayataaa in tha bahavioral aciancaa aa coaparad to coa- 
putar acianca and anginaaring. 

Thaaa raaulta aight appaar to auggaat that ainca thara ara far 
fauar F.A.a than P.A.a* tha affacta of raaolving F.A.a aay not ba 
aa larga aa a casual atudy of P.A.a uould indicata. It ahould ba 
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not«df houtvart that ainca diacouraa anaphora ara uoad by tha 
uritar to avoid naadlaaa rapatitiont anaphora ara aora likaly to 
ba uaad to raplaoa tha aajor ooncapta in a piaca of taxt. Aa a 
raaultf raaolvino avan tha aaan of 3.67 F.A.a par abatract aay 
hava a atrongly diffarantial iapact on tara fraquanciaa and ulti*- 
aatalyt ratriaval raaulta. Alaot ainca aoat piacaa of taxt ara 
organizad around ona or two aajor ooncapta* tha affact of laaving 
anaphoric rafarancaa untraatad haa tha oaaa potantial for auba^ 
tantiva iapact on any of tha inforaation acianca araaa uhich daal 
uith naturally occurring taxta* 



Tybla H Diatribution acroaa 2 Subjact Ooaaina 
P.A.a F.A.a 
Ko. Haan No. Raan 

PaycINFO 3960 13.2 1347 4.49 

I NSPEC 3024 10. 08 857 2. 86 

Total 6984 11.64 2204 3.67 



A Functional Indax (F. I.«»F.A/tP.A. ) uaa coaputad for aach 
claaa in aach databaaa (Tabla 2). Appandix A containa thia 
inforaation for aach individual tara in tha aat of 600 abatracta 
and Appandix Bt figuraa for tha 487 docuaanta uead in tha 
ratriaval axpariaant. Ihm F.I. ia an iaportant paraaatar of 
conaidaration aa ua ara intaraatad in davaloping raaolution 
algorithaa for thoaa claaaaa in »hich a high proportion of tha 
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P.A.8 mrm F.A.s. Givtn only th« inforaatfon in Table 2* aoaa 
claaaaa appaar far likaliar candidatas than othara. Fot axaaplat 
of tba 493 uaa3 of cantral pronouna? 78X of tha occurrancaa uara 
anaphoric. Thia high F.I. contraata uith tha iiaa of tha dafinita 
articia *tha' uhich haa a vary high fraquancy of occurranca (3435 
uaaa acroaa both databaaaa) yat an F.I. of only 14S. On thia 
baaia* it uould ba unlikaly that ona would chooaa to davota ona*9 
afforta to devaloping algorithaa for claaaaa uith ao lou an F. I.. 
Yat tha raaulta of tha ratriaval axparieant uill alao ba takan 
into conaidaration uhan chooaing claaaaa for algoritha davalop- 
aant. It aakaa aanaa to concantrata our afforta on thoaa claaaaa 
uith both a high F.I. and daaonatratad poaitiva affact on 
ratriaval parforaanca. 

ffV^f-GovfrT^etf Rffcoqn^fon pf fynctionyt Anaph^ff 

Tha othar araa of raaulta to ba raportad ia that of tha axtant 
to uhich tha rulaa for daciding uhan a P.A. ia an F.A. can auc* 
caaafully ba appliad by indapandant judgaa. Thaaa raaulta pro- 
vide praliainary avidanca of uhathar tha anvironaant in uhich an 
anaphoric uaaga occura ia pradictabla anough to aaka autoaatic 
racognition poaaibla. Thraa judgaa uara uaad for taating aach 
aat of rulaa. Tha judgaa uara not auarc that thair daciaiona 
uara on tha anaphoricity of a tara. Thay uara inatructad to fol- 
lou a aat of rulaa uhich daacribad diatinct pattarna of uaaga of 
a tara and dacida uhich pattarn a particular inatanca aatchad. 
Tha rulaa uaad by tha judgaa uara batiad on tha linguiatic ragu- 
laritiaa obaarvad and oapturad in individual analyaaa of aach 
functioning anaphoric tara and ara contained in Appendix C. 
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Tiibl» 2 t CUm Sunary 

PsycIMFO IKSPEC Totals 





Ana. 


Non. 


F.I. 


Ana. 


Non. 


F.I. 


Ana. 


Non. 


F.I 


Cantral Pronouna 


244 


60 


.80 


143 


46 


.76 


387 


106 


.78 


Noainal Dsaonstrativas 


176 


265 


.40 


155 


148 


.51 


331 


413 


.44 


Ralativa Pronouns 


227 


255 


.47 


192 


88 


.69 


419 


343 


.55 


Noainal Subetitutaa 


60 


63 


.49 


64 


71 


.47 


124 


134 


.48 


Pro-varb 


21 


42 


.33 


3 


12 


.20 


24 


54 


.31 


Indafinitas 


128 


317 


.29 


44 


209 


.17 


172 


526 


.25 


Pro-ad j actives 


27 


51 


.35 


10 


40 


.20 


37 


91 


.29 


Pro-advarbials 


25 


52 


.32 


30 


68 


.31 


55 


120 


.31 


S t( Ss 


188 


25 


.88 


0 


0 




188 


25 


.88 


Dafinita Articla 


251 


1483 


.14 


216 


1485 


.13 


467 


2968 


.14 


Totals 


1347 


2613 


.34 


857 


2167 


.28 


2204 


4780 


.32 



Tht rult mmtm consistftd of an ordarad flttriM of patt«rn aiittch- 
ing tasks against sithar syntactic or laxical taaplataSa Judgas 
dacida uhather a uaaga aatchas Rula It or Rula 2f and so ont do»n 
tha Hat of rulaa for that olaaa or tara. Soaa of thaaa rulaa 
dafina anaphoric xmmm* othara nonanaphorict but tha judga uaa not 
concarnad vith thia* Tha Judgas* daciaiona uara atrietly govarnad 
by tha pattarn satching aapact of tha rulaa. Thaaa ara tha typaa 
of huaan daciaiona that algorithas ara abla to aiaic and which 
will aaka tha autoaatio racognition of anaphoric uaas of taraa 
poaaibla* Tha avantual autoaation of thia taak would raquirat in 
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addition to tte algorithMt tb« inclusion of tuo coaponants coa* 
aonly avaliabia in taatt procaaaing ayataaat naaalyt a paraar and 
a laxicon aith aaaantio elaaa inforaation. (Saa (133 for a aaa- 
pla aat of rulaa for tba Koainal Oaaonatrativa 'that*.) 

Tabla 3 praaanta tha avaraga rata for olaaaaa l^t acroaa 
thraa judgact of corractly applying tba rulaa for daeiding which 
pat tarn of uaaga a particular inatanea of a tara folloua. Appan* 
dix 0 containa tha aueeaaa rata of applying rulaa for aach judga 
and ouaulattvaly for aach tara taatad. Tha auccaaa of tha 
pattarn-aatching rulaa in eorractly pradicting tha saaa daciaion 
aa an ovart intallaetual daciaion on a tara*a anaphoricity rangad 
froa a lov of 83> for tha taraa coapriaing tha S Ic Sa claaa to a 
high of 99S for tha provarb *do*« Thaaa initial raaulta giva ua 
confidanca in tha fiald*a ability to davaiop P.A*-to-F*A. algor-* 
ithaat particularly ainca an arror analyaia baa idantifiad tha 
racurring problaa aith tba rulaa to ba a difficulty in daeiding 
whan a aubaaquant dafinita noun phraaa containing a claaa laval 
noun rafara to tba aaaa antity aa a pravioua apacific noun (a.g. 
*tha inatruaant* uaaa tba P. A. *tba* plua a claaa laval noun uaad 
aa a laaa apacifiad rafaranca to a particular taat inatruaant 
aantionad aarliar in taxt). Inclusion of aaaantic claaa inforsra* 
tion in tba ayataa*a laxicon could aaaily laaaan tha nuabar of 
arrora of thia aort. 



Diacuaaion 

9itb a aaan occurranca of 3.67 functioning anaphora par 
abatract acroaa tba full aaapla of 600 abatractat thia atudy 
indicataa that taraa capalUa of anaphoric rafaranca occur auffi* 
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TabU 3 1 TMting of RulM 



1. Cttntral Pronouns 981 

2. NoBinal teaonstrativM 871 

3. R«Iativ« Pronouns 93* 

4. Noainal SuJbstitutM 88S 

5. Pro-Vsrb 99* 

6. IndsfinitM S9« 

7. Pro-adjsctivM 86f 

8. Pro-adv«rbiaIs 96X 

9. S It Sa 83« 



ciantly fraquantly in abatracta* to raiaa ({uaationa aa to tba 
adaquacy of tachniqiiaa Hhich uaa aurfaca counta of a tara aa a 
aufficiant aaaaura of tba total tiaaa that a concapt is rafarrad 
to in an abatract* In that anaphora tand to ba uaad for abortan- 
ing tha rafaranca to tba aajor concapts of a taxt* it ia intui- 
tivaly claart although aoaiting aapirical proof* that raaolution 
of tbasa anaphoric rsfarancss sill gsnarata tara fraquancias 
ohich provida battsr raprasantationa of tha inforaation contant 
of docuaants and iaprovs ratriaval in an oparational aattlng. 
Thaaa rapraaantstions will ba basad on tha fraquancy of rafaranca 
to a concapt rsthar than tha currantly uaad fraquancy of occur- 
ranca of a tara* 

In tha aacond phass of this raaaarch projact* sa conductad an 
axpariaant on tha iapact of raaolving anaphora in ona araa cf 
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inforiation aeicncttr iWMly bibliographic ratrittval* but eonaidttr 
thm mttmet to bm motm far rangino than juat ratriavalt aapacially 
•inea tha nuabara of F.A.a ia a function of tha langth of tba 
taxt. Any work iiith naturally occurring taxt ia affactad by tha 
linguiatic phanoaanon of diacouraa anaphora. Aa notad abovat work 
in tha araaa of quution-^anauaring* autoaatic axtractingt and 
quary analyaia hava acknoaladgad tha naad to davaiop tachniquaa 
for handling anaphoric taraa. Va ara hopaful that our raaulta 
aill provida aoaa pravioualy unavai labia baaa-lina data on dia- 
couraa anaphora in ona particular taxt-typa acroaa two aubjact 
doaaina. 

Raaulta of tha rula taating indicata that algorithaa for 
dataraining autoaatically ohathar a potantially anaphoric tara ia 
functioning aa an anaphor in a particular inatanca ara indaad 
faaaibla ainca tha taak haa baan ahoan to ba ona of pattarn 
aatching govarnad by rulaa appliad oith high raliability. In 
addition* a aiailar algorithaic approach for raaolving function- 
ing anaphora oith thair appropriata rafarranta «ill ba auggaatad 
for aavaraX of tha claaaaa of anaphora aftar a full analyaia of 
tha ratriaval axpariaant raaulta ia coaplatad. 
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THE EFFECTS OF ANAPHORIC RESOLUTION 
ON RETRIEVAL PERFORM ANCE CI] 



Jmtttmy Katzsr* Suean Bonzi* ElizalMth Liddy 

Syracuse Univarsity* School of Inforaation Studii 
Syracuaat Nav Tork 13244 



INTROOUCTtON 

For alaoat thirty yaara* work in autoaatic indaxing haa 
baan a aajor eoaponant of inforaation ratriaval raaaareh. 
To parfora affactivaly* indaxing achaaaa auat ba abla to 
aecurataly portray what tha docuaant ia about and thay auat 
aaaiat in tha diacriaination aaong docuaanta in tha 
collaction. Vithin docuaant f raquancy CVDF) ia claarly 
halpful in aaating tha firat of thaaa functiona: tha aora 
oftan a tara ia tiaad in a doctiaant* tha graatar tha 
likalihood that tha concapt or aubjact undaxlying tha tara 
ia central to tha docuaant. Houavar* it ia not claar that 
WDF ia of aueh valua to tha aacond function of indaxing 
achaaaa* For a tara to diatinguiah aaong docuaanta* ita VDF 
auat hava a larga varianca ovar tha collaction. Gi van that 
docuaanta ara eoapoaad of ralativaly faw uorda* auch aa 
titlaa and abatracta* eouplad with tha rathar aachanical 
aaana for autoaatically racognizing a gxvan tara (a.g. 
counta of aynonyaa ara uaually not coabinad)* it ia doubtful 
that any aizabla varianca in VDF would occur. Soaa aupport 
for thia contantion ia provided by Sparck Jonaa C2] uho 
found that 62S of tha taraa in ona aaall databaao had a tfOF 
of ona and another 19% of the teraa had a VDF of two. 
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Thertt tm littl« •vid«nc« of thm •ff«ct of incrMsing 
thm variation of VDF on ratriaval parforaanca. It la not 
anough to aiaply incraaaa tha VOF uaighta* To bm of valua 
in ratriaval tha incraaaa auat ba diaproportional* raiaing 
tha variability by affacting kmy taraa aora than othar 
taraa* Uaing longar docuaanta (full taxt inataad of 
abatracta) ia ona aathod to accoapliah thia* Anothar ia to 
bring togathar into ona claaa* all aantiona cf a aingla 
concapt — • uhath^r rafarrad to by tha aaaa root* by a 
aynonya* or by tha linguiatic tachniqua knoyn aa anaphora* 
Staaaing of auf f ixaa ia coaaon to aoat approachaa to 
autoaatic indaxing and thaaauri hava baan taaad to coobina 
aynonyaa into a aingla claaa. Houavart tha af fact of 
anaphora on VOF and ultiaataly on ratriaval parforaanca haa 
not baan atudiad* 

AKAPHORA 

Anaphora* briafly dafinad* ia tha linguiatic davica of 
abbraviatad aubaaquant rafaranca* Conaidar tha following 
aantanca 13 It 

tfaah and cora aix baking applaa and placa thaa in a pan. 

Tha p r onoun * ** thaa ** i a an anapho r and i a aaa i 1 y 
undaratood by paoplo to aaant **aix uaahad and cored baking 
applaa*** 

1* Thia raport ia baaad on a papar praaantad at tha 1986 
ASIS Annual Haating* Chicago* Illinoia. 
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Anaphora is ona of aavaral ao-callad cohaaion davicaa 
uaad in both urittan and apokan diacouraa to <1) avoid 
aonbtonoua rapatition* (2) ahortan tha diacouraa* and <3) 
anhanca tha coharanca of tha paaaaga. Bacauaa anaphora ara 
uaad to aliainata rapatitiouanaaa* thay ara aora likaly to 
ba uaad to raplaca tha aajor concapta and taraa in an 
abatract. Thua* ua would axpact that raaolving all anaphora 
in an abatract will incraaaa tha VOF of iaportant taraa 
proportionally aora than it will raiaa tha fraquanciaa of 
othar taraa* 

Although huaan intallact haa no difficulty racognizing 
and r aao 1 v i ng anapho ra ( rap 1 ac i ng t haa a i th thai r 
rafaranta)* autoaatic aathoda to accoapliah thaaa taaka ara 
atill in thair infancy. Vork in natural languaga 
undaratanding haa aada aoaa advancaa in tha traataant of 
anaphora* but that work ia raatrictad to liaitad aubjact 
doaaina or cartain claaaaa of anaphora. C 4-101 In 
inforaation acianca* anaphora ia alaoat coaplataly ignorad. 
Thara ia aoaa aantion of it in tha litaratura Cll-143* but 
not in taraa of docuaant ratriaval. Inataad* anaphora ia 
conaidarad in tha traataant of qxsaation-anauaring ayataaa* 
paaaaga ratriaval* or autoaatic abatracting. 

Tabla I praaanta tha aajor claaaaa of anaphora uaad in 
tha currant atudys aaa C151 for a aora coaplata daacription. 
Vhathar or not a givan anaphor actually functions 
anaphorlcally can only ba datarainad by analyzing tha 
linguiatic contaxt within which tha tara axiata. Thua* tha 
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CLASSES OF ANAPHORA 



ANAPHORIC CLASS 



EXAMPLES 



a: CENTRAL PRONOUNS 



th«y» thsir* thMMlvM 



B: NOHINAL OEHONSTRATIVES 



this* that* these* those 



C: RELATIVE PRONOUNS 



who* nhich* where 



O: NOniNAL SUBSTITUTES 



above* former* one 



E: PRO-VERBS 



do 



F: INDEFINITES 



aoae* all* each 



G: ADJECTIVES 



another* both* identical 



H: ADVERBS 



so* such* siatlarly 



i: SS 



(subj acts) 



j: DEFINITE ARTICLE 



th< 
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rMOlution task dapands upon (1) an •xhaustivs list of mil 
potantial anaphora* (2) a sat of rulas to dataraina if a 
particular potantial anaphor ia actual ly functioning 
anapborical Iy» and (3) a sat of rulaa for raplacing tha 
functioning anaphora uith thair rafaranta. 



HETHOO 

In thia atudy* ratriaval parforaanca dapanda upon tha 
dagraa to uhich tha pradictad ralavanca of ^'unraaolvad** and 
"rcaolvad" docuaants aatchas tha uaar^s ralavanca judgaants. 
Thus* thraa ssts of Judgaants ara naadad: (1) thoaa basad 
on tha usar^s assaasaant of docuaants ratriavad by an IR 
ayataa in rasponsa to a quary* (2) thoaa producad by tha 
ratriaval systas fros unraaolvad staaa in tha docuaant* and 
(3) thoaa producad by tha ayataa froa tha raaolvad ataaa in 
tha docuaant. 



Databaaaa* Ouarias* k Halavsnca Judoaanta: Sinca tha thraa 
ralavanca Judgaanta notad abova can ba producad in a 
poat-ratriaval axpariaent* quarias and ralavanca Judgaants 
collactad in othar studiaa could ba ra*analyzad for our 
currant work on anaphora* Only a briaf dascription of thaaa 
axiating aatarials will ba providad hara: a ful lar 
accounting can ba found alaauhara. C163 
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Two databasM uara uaad to incraaaa tha ganarality of 
tha findinga* Each uaa coapoaad of approxiaataly 12*000 
docuaanta conaiating of a citation and an abatract of 75*^175 
Engliah uorda. Froa tha aarlier atudy» ua had 04 quariaa to 
tha INSPEC databaaa and 57 quariaa to PaycINFO. All quariaa 
uara poaad by individuala with ganuina inforaation naada and 
uara aaarchad by trainad intaraadiariaa* Tha ralavanca of 
tha ratriavad docuaanta uaa datarainad by tha originator of 
tha quary uaing a four-point catagorical acala: 1 baing 
highly ralavant* 2 alightly ralavant* 3 alightly 
non'-ralavant* and 4 highly non-ralavant. 

Tha currant raaaarch could aaka uaa of only a aaall 
aubaat of tha avai labia quariaa. Soaa quariaa had to ba 
axcludad bacauaa thara uaa inaufficiant variability in tha 
ralavanca judgaanta aaaignad to tha ratriavad docuaanta* 
Gthara uara axcludad to dacraaaa tha aaount of uork involvad 
in idantifying and raaolving "by hand" all anaphora in all 
ratriavad docuaanta. Quariaa uara aalactad uhxch aat tha 
follouing critaria< (a> tha nuabar of ratriavad docuaanta 
rangad batuaan 15**30» Cb> thara uara at laaat tuo ratriavad 
docuaanta judgad at aach of tha four ralavanca catagoriaa* 
and (c) no aora than 60% of tha ratriavad docuaanta uara 
judgad ralavant — in catagoriaa 1 or 2* Thaaa critaria 
aalactad 12 quariaa froa INSPEC and 17 froa PaycINFO. Fiva 
quariaa uara randoaly diacardad froa PaycINFO to aaka tha 
tuo aata aqual. Tabla 2 daacribaa thaaa tuo taat 
collactiona* 
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TABLE 2 

DATABASE POPULATIONS AND SAHPLES 



AVAILABLE DATA 



INSPEC PsycINrO 



NUHBER OF DOCUHENTS 



12.864 



11.662 



NUHBER OF USER QUERIES 



84 



52 



NUHBER OF TYPES IN DATABASE 



67.401 



35.758 



SAMPLE USED 



NUHBER OF QUERIES 12 12 

DOCUHENTS RETRIEVED BY EACH QUERY 12 15-25 

UNIQUE DOCUHENTS IN ALL QUERIES 261 226 
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For Mch of thtt 487 rtttrittv«d docuMntfl all potantial 
anaphora uara idantifiad by coaparing aach tara in tha fila 
of docuaanta with a **dictionary** of all anaphora that 
occurrad in tha tuo databaaaa. Ovar 6500 potantial anaphora 
uara found. Each of thaa .uaa inapactad within ita 
linguiatic contaxt to dataraina if it actually functionad 
anaphorical ly in tha docuaants ovar 2200 trua anaphora wara 
idantifiad. Tha final atap waa to apply anothar aat of 
rulaa to raaolva all functioning anaphora. Tha dictionary 
of anaphora and tha rulaa to diacriainata batuaan potantial 
and functional anaphora wara davalopad and validatad on 
othar docuaant aaaplaa froa tha two databaaaa. CIS! At thia 
point* two collactiona of tha 487 docuaanta axiatads ona aa 
originally containad in tha databaaa and ona with all 
anaphora raplacad with thair rafaranta. 

Tara-waiQhtina Sfehaaaa snd $4aiU«^^tY StSSMICK* Dif farant 
aathoda for uaighting taraa and for dataraining tha dagraa 
of aiailarity batwaan docuaanta and tha quary af fact 
ratriaval parforaanca diffarantly. C17} Tharafora* it waa 
iaportant to conaidar altarnativa approachaa to 
tara-waighting and aiailarity« Tabla 3 liata tha 12 
tara-waighting achaaaa aaployad in tha atudy. float of thaaa 
includa VDF — aithar alona* corractad for langth* or in 
eoab ination with collaetion f r aquanc i aa * Co 1 1 ac t i on 
fraquanciaa and tara poatinga wara tabulatad aaparataly 
uaing alightly dif farant aathoda of procaaaingt for 
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TABLE 3 
TERH-VEIGHTING SCHEHES 



(a) 


1 


(g) 


f/log(k) 


(b) 


l/t 


(h) 


f/F 


(c) 


l/log(t) 


(j) 


f/log(F) 


(d) 


f 


(1) 


f/t (k) (F) 3 


(e 


log(f) 


(a) 


f/C log(k) (F) ] 


(f) 


f/ (k) 


<n) 


C f IC Iog(N/d) ] 




d - nu«b«r of posting* ot tmrm 
f = uithin docuMnt frequency; 
F » frequency of tere in dmtebaae; 
k ^ nueber of tokens in docueents 
M » nueber of docueente in database; 
t = nueber of typee in docueent. 
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approximately aight parcant of tha taraat tha nuabar of 
poatinos uaa highar than tha col lac t ion fraquancy. Thaaa 
dif farancaa pravantad our uaa of any tara-uaight that 
coabinad with poatinga and collaction fraquanciaa. VOF» 
houavar* uaa not affactad by thaaa diffarancaa. Tha coaina 
correlation and Dica^a coafficiant uara tha two aiailarity 
aaaauraa uaad. Coabining tha tara uaighta with tha 
aiailarity aaaauraa yialdad 19 diffarant paira. « 
^nalY^ ie: Tha aajor raaaareh quaation can ba anauarad by 
coaparing tuo atatiatical ralationahipa: (1) that batuaan 
tha uaara* ralavanca judgaanta and ayatea'a ralevanca baaad 
on unraaolvad anaphora* and (2) that batuaan tha uaara' 
judgaanta and tha ayataa'a ralavanca baaad on raaolvad 
anaphora. If tha aacond relationahip ia atrongar than tha 
firat* it aay ba raaaonabla to conclude that raaolving 
anaphora in a docuaant will affact VOFa in auch a uay ao aa 
to iaprova retrieval perforaance. 

There are thouaanda of relaticnahipa to ba ^oapared. 
Each coabination of tera-ueighting achaae and aiailarity 
aaasura uaa uaed aeparately on each of the ten claaaea of 
anaphora (aee Table 1) and on an ''eleventh** claaa» aade up 
of the union of the other ten. Thie entire eet of analyaee 
uaa carried out for all queriea in the tuo databaee. Each 
relationahip uaa quantified uaing Pearaon'a uell-knoun 

1^ Vith the Coeine Correlation* tare uaighte b and c 

are equivalent to at f and g are equivalent to d; 
and L ia the eaae aa H. 
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Mmsurfi oS llnMr eorr«latlon« Th« two corralations 
(b«tiiMn thm u««r«* JudgMnts vsraus rMolv«d doeuMnts and 
ummrm^ judgMnts vatflus untMoIvad doeuMnts) umtm eoaparad 
to mmm it onm im 0tatliitlcmlly hlghar than tha othar. Tha 
analyala plan ia auaaarisad in Tabla 4. 
RESULTS 

Claarly* aith ovar 5000 eoabinationa of raaulta to 
conaidar* it ia ditficult to draw aiapla concluaiona. 
Horaovar* cara auat ba ta^an in intarprating individual 
findinga baeauaa atatiatically aoaa 250 taata (of 
diffarancaa batuaan tha two corralationa) could achiava 
aignificanca at tha •OS laval by chanca alona. Tharafora* 
tha ganaral pattarna of raaulta ahoun in Tabla 5 and on 
Appandix F uill ba axaainad rathar than tha rau findinga 
givan in Appandix E* 

In ganaral* tha raaulta ara aixad* For tha aajority of 
quariaa* raplacing anaphora with thair rafaranta did not 
hava any raal Cnon-ehanca;) affact on tha pradictad ordar of 
doc uaan t r a 1 a vanoa • 

Soaa raaolutiona had a nagativa affact* i.a. raaolving 
anaphora raducad tha ratriaval parforaanca in taraa of 
ranking. Tha aoat obvioua axaapla of thia ia INSPEC Quary 
#109 which had nagativa raaulta in four diffarant claaaaa of 
anaphora. Tha aoat likaly axplanation for nagativa findinga 
aay ba docuaant langth« Raaolving anaphora doaa not aiaply 
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TABLE 4 



ANALYSIS 



For a givan quaryt TW* and SH* do 



UNRESOLVED RANKINGS 

1. Rank docuaanta by pradictad ralavanca 

2. Corralata thaaa ulth uaar*a ralavanca 



RESOLVED RANKINGS 

3. Raaolva a aingla claaa of anaphora in 

all docuaanta 

4. Racoaputa TVa and Slla 

5. Rank docuaanta by pradictad ralavanca 

6. Corralata thaaa with uaar'a ralavanca 



COHPARE •2 vc. as 

7. Oataraina which aat of rankinga battar 
aatch tha uaar*a Judgaanta 



REPEAT ALL OF THE ABOVE FOR 

A. All coabinationa of TVa and SHa (19) 

B. All claaaaa of anaphora and a 

coabinad claaa (11) 

C. All quariaa (12) 
0. All databaaaa (2) 
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r^plactt m singltt uork (such as a pronoun) uxth anothsr 
siapls uord (such as ths noun to which ths pronoun rsfsrs). 
Instsad* anaphora say naad to bs rasolvsd with phrasss of 
ssvsral words — sost of which can bs trivial. Sines soss 
of ths tsrs-wsiQhtino schsaas and ths sisilsrity asaeurss 
wsrs not corrsctsd for docuasnt Isngtho rasolution could* in 
thcss casss* havs had a nsgativs offset. 

Howsvsrt it is also svidsnt fros Tabls 5 thst 
rssolution incrsassd rstrisvsl psrforssncs for ssvsral 
qusriss — #158t ttlSOt •203* snd •212 ssss sost obvious. It 
is worth noting that positivs sf fscts for ssvsral anaphoric 
classss do not nacsssarily sceusulats into an ovsrall 
positivs sffsct whsn all elassM ars rasolvsd (Class R)l for 
•158 thsrs isn't any ovsrall sffsct* whils for ^212 ths 
ovsrall sffsct is sixsd. Thsrs is no clssr psttsrn of whst 
is rsquirsd to obtain a positivs rssult in Class R — 
coapars qusry SIOT with #170* or #221 with •222. Obviously* 
total rssolution (Class R) is a cosplsx phsnossnon* ons 
aspsct of which is liksly to bs docuasnt Isngth. 

Looking st ths othsr classss of anaphora rsvsals littls 
bscauss* in gsnsral* fsw clsar pattsrns sasrgs. Only two 
classss producsd consistsnt positivs rssults in both 
dstabassst ths nosinal substitutss (D) snd ths sdvsrbs (H). 
No class of snaphora producsd cosparabls nsgativs findings. 
For ths csntrsl pronouns (Clsss A)» ths diffsrsncss srs 
bstwssn ths two databasss. Enginasrs do not ssss to uss 
thsss pronouns as oftsn or in ths saaa aannar as do writsrs 
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in thm social/bahavioral seiMCM. For INSPEC* not a ainola 
quary was affactad* poaitivaly or nagativaly* by raaolving 
thaaa pronouna. Vharaaa for PaycINFO* thraa quariaa 
profitad froa tha raaolution of pronouna and nona uara 
advaraaly affactad by it. 

Thara ara othar diffaraneaa aaong tha databaaaa. 
Appandicaa E and F ahoua that PaycINFO had tuica aa aany 
poaitiva finding* aa INSPEC* but both had approxiaataly tha 
aaaa nuabar of nagativa findinga* In Tabia 5» wa can aaa 
diffaraneaa in taraa of quariaa* Though tha 12 quariaa froa 
aach databaaa uara aalactad carafully* thraa froa INSPEC 
(«142t #l82t «184)9 but only ona froa PaycINFO (#223) had no 
aignificant raaulta in any anaphoric claaa. Thaaa 
diffaraneaa batuean tha databaaaa ara probably raal and 
raflact raal diffaraneaa in tha uriting atyla of aaeh fiald 
and tha natura of ita vocabulary* 

I n auaaary • tha raaul ta i nd i eata that a dirac^ 
aubatitution of anaphora uith thair rafaranta la not likaly 
to iaprova ratriaval parforaanca of aciantific abatracta. 
tnataad* if anaphora ia to ba uaaful in ratriaval 
af factivanaaa* it will hava to ba traatad in aoaa aora 
eoaplax aannar than uaa attaaptad hara. Soaa obvioua 
traataanta ara diacuaaad balou* 
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TABLE 5 

SUIfRARY OF STATISTICAL RESULTS* 
ANAPHORIC CLASS 



INSPEC 

QUERIES ABCOEFGH 



I-lOl 

1-103 ♦ 
1-104 

1-107 ♦ ♦ 

1-109 - - - - 

1-135 

1-142 

1-158 ♦♦♦♦♦♦ 

1-170 - ♦ - 

1-180 ♦ ♦ ♦ ♦ ♦ ♦ 



1-182 
1-184 



PsyclNFO 
QuarlM 

P-203 ♦ ♦ ♦ ♦ 

P-207 ♦ 

P-212 •»-•♦••♦•♦ •♦•/- ♦/- 

P-219 ♦ ♦ 

P-221 ♦ - <. 

P-222 

P-223 

P-227 ♦ 
P-230 

P-235 ♦ ♦ 

P-248 ♦ - ♦ 

P-252 - ♦ ♦ ♦ - 



ttSign indieatM prsMnca of at l*ast ona findino 
that raaolution aignificantly affactad 
(poaitivaly or nagativaly) ratriaval parforaanca. 
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Fmgm 35 



DISCUSS I OH 

This studly uss basad on uhat atiil smm to ba a 
plaiMlbla in a docuaant and tharafora* by raplacing th«r 
with thair rafaranta tha VOF of laportant taraa iilll ba 
raiaad diffarant tally in coapariaon with laaa iaportant 
taraa. Though ua ara plaaaad to find aoaa raaulta uhich 
aupport that hypothaaia* ua hava not* aa yat* baan abla to 
axplain why no changa uaa found in tha aajority of quariaa 
atudiad. Ifor hava ua baan abla to dataraina why* for aoaa 
of tha quariaa* tha raaulta tiara eountar to tha hypothaaia. 

Docuaant langth ia ona intaraating poaaibility for tha 
anoaaloua raaulta. Abatracta* aa ralativaly abort 
docuaanta* aay contain too fau anaphora to affact a aiaabla 
changa in VOF aftar raaolution — thara ia only an avaraga 
of 4.5 anaphora in PaycINFO and 2*9 anaphora in INSPEC. 
Parhapa tha raaolution of anaphora uill prova aora affactiva 
on longar doctjaanta auch aa thoaa found in full-taxt 
ayataaa. Furtharaora* aa notad aarliar* tha raaolution 
procaaa fraquantly incraaaaa tha langth of tha docuaant — 
oftan uith non-^ubatantiva taraa* Thaaa factora* coabinad 
with tha aanaitivity of tha ranking aathoda to docuaant 
langth* aay account for aany of tha raaulta which ran 
eountar to our hypothaaia. 

To axplora tha affact of docuaant langth* two furthar 
analyaaa can ba conductad uith tha axiating data* Firat* 
othar ranking aathoda can ba triad* onaa not baaad on tara 
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Pmam 36 

ualghting achMM or siailarity aaasurM that mrm ••nsitiva 
to thm numbmr o t t okons i n thm docutton t • Socond • 
rMolutiontt can bm autoMtlcally coaparad with quary taras 
to anaura that only auJbatantiva taraa ara addad to tha 
raaolvad vara x on of tha docuaant* tfhathar thaaa analyaaa 
ahiid light on tha varioua aapacta of docuaant langth raaaina 
to ba aaan« 

Anothar poaaibla contributor to tha unanticipatad 
raaulta is tha fora of tha ralavanca jtxtgaant* A aora 
continuoua aaaaura of ralavanca uould hava givan aora pouar 
and aanaitivity to tha atatiatical aaaauratf* Vhan tha 
diffaranca batuaan tha ralationahipa baing taatad in 
Appandix E ara not atatiatical ly aignificant* it aay ba 
bacauaa thara ia no affact on raaolution* Houavar* an aqual 
ralationahip can alao occur uhan ganuina diffaranca* axiat. 
Bacauaa tha uaara* ralavanca judgaanta uara originally 
collactad on a groaa acala* aaaauraa of ralationahip ara 
inaanaitiva to diffarancaa in pradictad ralavanca within any 
ona of tha four iiaar-givan ralavanca catagoriaa* Thua aoaa 
of tha anoaaloua raaulta could ba cauaad by a aaaauraaant 
liaitation* Though poaaibla* ua find thia axplanation laaa 
plauaibla than that of docuaant langth* 

Othar axplanationa for tha obtainad raaulta ara likaly 
to aaarga f roa a caraful atudy of individual ratriavad 
docuaanta. It ia Hkaly that aoaa docuaanta ara atrongly 
affactad by raaolution uhila othara ara not* Thia atudy 
axaainad tha affact of raaolution againat a quary and aa a 
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tMult "avarao**^ out" th« •£f«ct on th« individual 
docuB«nt«. A thorough analy«i« of what happened to 
individual docuaanta within a givan quary ahould ba 
inatructiva. 

Froa Tabla 5 aavaral quariaa aaaa obvioua candidataa 
for thia "aicro-avaluation" C181. Quary #109 ia intaraating 
baeauaa all of tha significant findings uara nagativa and 
thara was no cuaulativa affact in Class R. Quary ttlSS is 
siailar axcapt that tha raaulta for tha individual anaphoric 
cla aa a a wara poaitiva. tt aight ba inatructiva to coapara 
tha analysis of aiSS with that of »iao (and ^202) uhara all 
tha positive raaulta did laad to an overall poaitiva effect 
in Claaa R. Query 11212 ia the only query that produced 
aixed results in the aerged reaolution eat; perhaps 
aoaething could be learned froa it. Finally, it probably 
would be uaeful to exaaine a couple of queriea that failed 
to achieve any aignificant raaulta after reaolution. Taken 
together* this sort of failure analyaia aay enable ua to 
coae to a final condition about the viability of our 
original hypothesis* or at laaat that vereion of it that 
pcrtaina to abstract-length docuaante. 



48 



Pag« 38 



REFERENCES 



1. Thi« raport im basad on m papar praaantad at tha 1986 
ASIS Annual Itaatins* Chicaoo* Illinoia. 



2. Karan Sparek Jonaa* "Indax Tara Vaighting"* Inforaation 
Storaaa and Batriaval. S (1973). 



3. n.A.K. Halliday* It Ruqaiya Haaan* Cohaaion In Enaliah* 
(Londonx Longaan Ciroup* 1976) • 



4. Danial G. Bobrou* "A Quaation Anauaring Syataa for High 
School Algabra Vord Problaaa*** AFIPS Confaranea 
?Iflff<ftft,ffQa> 26 (1964). 



5. Tarry Vinograd. Undaratandino natural Lanauaoa. (Nau 
Yorkt Aeadaaic Praaa. 1972). 



6. Villiaa A. Vooda. at al. Ztat mNAR Seianea Matural 
^ngufot Inforaation Svataat Firal Raoortc (Caabridga* 
naaa: Bolt* Baranak and Nauaan* Inc.* Raport 2378* 
1972). 



7. Charlaa J. Riagar* "Concaptual Ilaaory and Infaranca". 
In Rogar C. Schank (Ed.)* Coneaotual Inforaation 
Proea— inn. (Aaatardaax North-Holland* 1975). 



8. Torick A. Vilka* "An Intalligant Analyzar and 
Undaratandar of Engliah*** Coaaunicat t ona of iSSL* 
(1975) 264-274. 



9. Donald E. tfalkar. Undaratandino Sookan l,^nQi|tffl^» 
(Aaatardaax North-Holland* 1978). 



10. Vandy Lahnart* JJjm, Procaaa fil Quaation ^nyufftngi ^ 
Coaauta r Siaulation l^oanition. (Hilladala. N.J.. 

Lauranca Erlbaua* 1978). 



11. Garard Saltan Ic llichaal J. HcGill* Introduetion ^ 
Wodarn Inforaation Ratriaval* (Nau York* IfcGrau-Hill* 
1983). 



49 



Prngm 39 



12. John O'Connor* **7ttxt SMrchino Rstrittval of Answer 

S«nt«ncM and Oth«r Answer Passagss*** Journal fiX ths 

Assrican Socf tv Inforaation gfrtyngf* 24 (1973) 
4A5-4tS0. 



13. Donald Valkart ^Thm Organization and Uaa of Inforaations 
Contributions of Inforaation Soisncs* Coaputational 
Linguistics and Artificial Intslligsncs**. Journal ^ 
iUft Aaarican Sociatv for Inforaation Sciancs> 32 C1981) 
347-363. 



14* Chris D. Paics* ^'Ths Autoaatic Gansration of Litsraturs 
Ahstractss An Approach Basad on ths Idantif ication of 
SslS-Indicating Phrasss**. In R. N. Oddy CEd. ) 
T^tPTli^lOT Wtrisval Basaarch* CLondons Buttsrworths. 
1981)* 172-191. 



15. Elisabsth Liddy* at al* **A Study of Discoursa Anaphora 
in Sciantif ic Abstracta*** Journal qX. tha Aaarican 
Soeistv Inforaation Scianca* in prass. 



16. Jsffrsy Katssr* at al. L Stxjdv of tha lanact gi. 
Raorassn tat ions In Inforaation Ratrisval SXa^SSM.* 
(Final Raport to tha National Scianca Foundation* July 
1982). 



17. Hichaal J. llcGill. at al. An Evaluation si. F»ctora 
Affactino Doeuaant Ra ^nkina b£ Inforaation Ratrisval 
Svstaasr (Final Raport to tha Nationsl Scianca 
Foundation* 1979) . 



18. 0. tf. King li E. C. Bryant* Tha Evaluation gJL 
In|orfgt^jt9T f S^ rY^^OT snd Products, (Vaahington* D.C. • 
Inforaation Rasourcas Praaa* 1971). 



50 



Page 40 



BIBLIOGRAPHY 



51 



Paga 41 



BrBLIQGHAPHY 

Balkint N.J.; Oddy* R.N. l Brookst H.R. "ASK for Inforaation 
Ratriavalt Part 1. Background and Thaory. Part lit 
Raaulta of a DMign Study." Journal si. Docuaantation. 38 
(2.3): 61-71. 145-1641 1962. 



Bobrou* Danlal G. t "A Quaation Anauarlng Syataa for High 
School Alabra tford ProbIaaa"t API PS Confaranca Procaadinoa* 

26 (1964). 



Griaaa* J. Thf Thraad of Diacouraa. (Tha Hagua: Houton 
Publiaharas 1975.) 



Halliday. R.A.K. • Ir Ruqaiya Haaan. Cohaaion in Enaliah 
(London: Longaan Group* 1976). 

Hirat. G. Anaphora in Natural Lanouaoa Undaratanding: A 
Survav. Ha» York* Springar- Varlag: 1981. 



Katzar. J., at al. A St\«lv of tha laoaet of Haaraaantationa 
in Inforaation Rat r lava 1 Svataaa. (Final Raport to tha 
National Scianca Foundation* July* 1982). 



King* O.V. te Bryant* E.C. Tha Evaluation gj. Inforaation 
Sarvieaa an^ Produeta. Vaahington* D.C. Inforaation 
Raaourcaa Praaa* 1971. 



Lahnart* Vandy* Hut Procaaa ftl Quaation Anawarinox 4 
Coaputar Slaulation £l Cognition. (HiUadala* N.J.. 
Lauranca Erlbaua* 1978). 



Liddy* Elizabath* at al. "A Study of Diacouraa Anaphora in 
SCiantif ic Abatracta". Journal Qt iha. Aaartean §s£lMl^ iSS. 
Inforaation Scianca* in praaa. 



HcGlll* Hichaal J.* at al. im Evaluation gj. Facta pa 
Affaetino Ooeuaant Ranking UUgrMtUn Wtrtaval Svataaa. 
(Final Raport to tha National Scianca Foundation* 1979). 



O'Connor* John* "Taxt Saarching Ratriaval of Anauar 
Santaneaa and othar Anauar Paaaagaa". Journal aX tha 
Aaartean Soeiatv for Inforaation Scianca* 24 (1973) 445-460. 



52 



P«0« 4 2 



Oddy. R.N. L Study qji R^prMmtat i ona lo£. Anoealoy §t»tf^ 
91 Kncmledoe in tnforMtion Bft^ifYPl* NSF Grant Proposal 
IST-8420608* Syracusa* Nau Tcrk: Syracuaa Univaraity* 
School of Inforaation Studiaat 1964. 



Paicat Chria 0* **Tha Autoaatic Ganaratlon of Lltaratura 
Abat rac ta> An Approach Baaad on tha I dant i f i cat ion o f 
Salf-Indicatino Phraaaa**. In R.N. Oddy (Ed.) Inforaation 
aatriaval Raaaarch* (Londont Buttaruortha* 1981 )• 172-191. 



Quirk* R. { Graanbaua* S. ; Laach* G. s Svartik* J. A Graaaar 
21 Contaanorarv finflUslL* London s Longaanai 1972. 



Riagar* Ch&rlaa J* ""Concaptual Haaory and Infaranca*** In 
Rogar C. Schank (Ed. ) ypnc^otual Inforaation Procaaaino. 
(Aaatardaa: North- Holland* 1975). 



Sal ton* (>arard ti RcGill* Hichaal J. Introduction to Hodarn 
Inforaation Ratriaval. (Nau York* HcGrau-Hill* 1983). 



Sidnar* C. Towarda Coanutat tonal T?)f9rY Sl Pt^l^t^t 
Anaphora Coaorahanaion in gnprliah Diaeouraa, Artificial 
Intal 1 iganea Laboratory Raport TR-537; Caabridga* 
Haaaachuaattat Raaaachuaatta Inatituta of Tachnologys 1979. 



Sparek Jonaa* Karan* **Indax Tara Vaightings* Inforaation 
Storaoa and Ratriaval* 9 (1973) 619-633. 



Valdatain* R. Tha Rola of Noun Phraaaa a£ Con tan t 
Indicatora. Ph.D. Diaaartation* Syracuaa Univaraity* 
School of Inforaation Studiaa* 1981. 



Valkar* Donald E. M»daratandiTiQ §P9HtT^ ItingWQf* 
(Aaatardaas North-Hol land* 1978) . 



Valkar* Donald* **Tha Organization and Uaa of Inforaation: 
Contributiona of Inforaation Scianca* Coaputational 
Linguiatica and Artificial Intal liganca**. Journal q1 tha 
Aaarican Sociatv for Inforaation Scianca* 32 (1981) 347-363. 

Vabbar* B.L. A. Po«-"l Approach is. OfffgUrtt AHMftgn* Nau 
Torks Garland Publiahing* Inc. I 1979. 

Vilka* Yorick A.* **An Int^lliant Analyzar and Undaratandar 
of Engliah*** CQaaunieationa of tha ACH* 18 (1975) 264-274. 



53 



Pag* 43 



Vinoorad. Tsrry. UndTata ndino Natural Lanouaoa. (Nau Yorkt 
Aeadaalc Praaa. 1972). 

Vooda. Villiaa A., at al. IfeS. LUMAR Seianea Matural 
Lanauaoa Inforaation Svataa Final Raport. (Caabridga. Haaa: 
Bolt* Baranak and Nauaant Inc.* Raport 2378* 1972). 



54 



APPENDICES 



Page 



Appendix A - Preliminary Test» 45 

Functional Indexes 

Appendix B - Retrieval Experiment 55 

Functional Indexes 

Appendix C - Linguistic Analysis 67 

Appendix D - Test Results of Rule Sets 131 

Appendix E - Retrieval Test Results 134 

INSPEC Series 100 
PsycABS Series 200 

Appendix F - Summaries of Statistical 354 

Results, INSPEC and 
PsycABS 



55 




APPENDIX A 



Prel iminary Test » 
Functional Indexes 
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PRELIMINARY TEST 



CLASS SUMMARY 





PSYCH. ABS. 




INSPEC 




TOTALS 


TERM 


Ana. 


Non. 


F.I. 


Ana. 


Non. 


F.I. 


Ana. 


Non. 


F.I. 


Central 
Pronouns 




60 


.80 


143 


46 


.76 


387 


106 


.78 


Nominal 
Demon strativ< 


\ 176 


265 


• 

.40 


155 


148 


.51 


331 


413 


.44 


Relative 
Pronouns 


227 


255 


.47 


192 


88 


.69 


419 


343 


.55 


Nominal 
Substitutes 


60 


63 


.49 


64 


71 


.47 




134 


.48 


Pro-verb 


21 


42 


.33 


3 


12 


.20 


9A 


54 


.31 


Indefinites 


128 


317 


.29 


44 


209 


.17 


179 


526 


.25 


Adjectives 


27 


51 


.35 


10 


40 


.20 


0/ 


91 


.29 


Adverbs 


25 


52 


.32 


30 


68 


.31 


55 


120 


.31 


S & Ss 


188 


25 


.88 


1 






188 


25 1 

1 


.88 


Definite 
Article 


251 


1483 


.14 


216 


1485 


.13 


467 


\ 

1 

1 
1 

2968 i 


.14 


TOTALS 


1347 


2613 


.34 


857 

57 


2167 


.28 

1 
1 

j 


2204 

t 

t \{ 


4780 

1 

1 

1 


.32 



"Central" Pronouns 
(Personal, Possessive, Reflexive) 



Appendix A^Z 



CLASS SUMMARY 



PRELIMINARY TEST SET 



TERM 



PSYCH. ABS. 



Ana . 



Non. I F.I . 



Ana 



INSPEC 



Non, 



F.I. 



Ana . 



TOTALS 



Non. 



F.I. 



CO 

Pi 
« 

I 
I 
I 



H 
CO 
CO 
H 
CO 
CO 

o 

P4 



H 



he 

him 

I 

she 

them 

they 

us 

we 



her 

his 

its 

our 

their 

your 



herself 
himself 
itself 
themselves 



0 
6 
12 
41 
0 
0 



12 
13 
16 

0 
90 

0 



1 
5 



2 

3 



1.00 
0 

l.OQ 
1.00 
1.00 

0 

0 



1.00 
1.00 
1.00 

0 

1.00 
0 



6 
1 
0 

mm 

3 
14 
0 
0 



0 
1 



7 
42 

0 
28 

0 



1.00 
1.00 
1.00 



1. 00 
1.00 
0 

1.00 
1.00 

0 

0 



1.00 
1.00 
0 

1.00 
0 



11 
1 

0 
6 
15 
55 
0 
0 



12 
20 
58 
0 

118 
0 



1 
6 



4 

3 



2 
3 



1.00 
1.00 
0 

1.00 
1.00 
1.00 

0 

0 



1.00 
1.00 
1.00 
0 

1.00 
0 



1.00 
1.00 
1.00 



Sub-total 
it 

TOTALS 



209 

35 

244 



12 
48 

60 



.95 
.42 

.80 



101 
42 

143 



3 
43 

46 



58 



,97 
,49 

,76 



310 
77 



15 
91 



387 i 106 



.95 
.46 

.78 



Nominal Demonstratives 



Appenaix a-j 



CLASS SUMMARY 



PRELIMINARY TEST SET 



TERM 



this 
these 
those 
that 



PSYCH. ABS. 



Ana. 



71 
61 
29 
15 



Non. 



6 

252* 



P.I. 



.91 
1.00 
.83 
.04 



INSPEC 



Ana. 



88 
50 
7 
10 



Non. 



F.I. 



59 

3 

1 
85* 



.60 
.94 
.88 
.10 



TOTALS 



Ana. 



159 
111 
36 
25 



Non. 



66 
3 
7 

337* 



F.I. 



.71 
.97 
.84 
.07 



TOTALS 



176 



265 



.40 



155 



148 



.51 



331 



413 



.44 



I 



J 



*occurrenctes 
included 



of tthat" used as 
in thir> figui e. 



a relative p 



ronoun 



ire not 
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Relative Pronouns 



Appendix A-4 



CLASS SUMMARY 
PRELIMINARY TEST SET 



TERM 



who 

whom 

^ose 

which 

where 

that 



PSYCH. ABS. 



Ana. 



39 
3 
3 

71 
9 

102 



Non. 



2 
1 

252* 



F.I. 



1.00 
1.00 
1.00 

.97 . 

.90 

.29 



Ana. 



INSPEC 



Non. 



2 

120 
19 
49 



2 
1 
85* 



F.I. 



1.00 

1.00 
.98 
.95 
.37 



Ana. 



TOTALS 



Non. 



41 
3 
5 

191 
28 
151 



4 

2 

337* 



TOTALS 



227 



255 



.49 



192 



88 



.69 



419 



343 



*occurren< 
included 



s of 
In this 



that" 
f iguije 



as a noidnal 



c emonst rative 



are not 



I 

f 

i 
» 
I 
I 

i 

I 



I 



60 



Nominal Substitutes 



Appenaix a>s 



CLASS SUMMARY 
PRELIMINARY TEST SETS 





PSYCH. 


ABS 


INSPEC 




TOTALS 




TERM 






















Ana. 


Non. 


P.I. 


Ana. 


Non« 


P.I. 


Ana. 


Non. 


F.I. 


above 




3 




1 




1.00 


1 


3 


.25 


former 








2 




1.00 


2 




1.00 


last 




2 


0 




2 


0 




4 


0 


latter 




A 
U 


1.00 . 


A 
H 


A 

u 


1.00 


/ 


0 


1.00 


one 

one's 

ones' 


7 


17 


.29 


8 


30 


.21 


15 


47 


.24 


other 


33 


8 


.80 


27 


4 


.87 


60 


12 


.83 


others 


A 
H 


D 


.40 


o 


A 


1.00 


1 A 

10 


6 


.62 


same 


1 1 

1 


t 


.46 




XX 


.26 


X7 


26 


.40 


Sub-Totals 


! 

DU 




.54 






.52 


XX ^ 




.53 


first 




D 


0 


Q 

o 


XZ 


.40 


Q 

o 


X / 


.32 


second 




2 


0 


•5 


7 


.30 




Q 


.25 


third 






0 


X 




1.00 


X 




.25 


fourth 


- 


1 


0 




2 


0 




3 


0 


fifth 




1 


0 




1 


0 




2 


0 


sixth 


- 








1 


0 




1 


0 

w 


seventh j 




















eighth 








_ 












ninth 








1 


1 


0 


1 
» 


1 


0 


tenth 








. 1 

1 






- i 

1 






Sub-Totals 




12 


0 


12 1 

» 

i 


24 


.33 


12 i 


36 


.25 


TOTALS 


60 

• 


63 I 


.49 


64 

61 


71 


.47 


124 


134 


.48 



Proverbial 



CLASS SUMMARY 
PRELIMINARY TEST SET 



TERM 



PSYCH. ABS. 



Ana. 



Non. 



F.I. 



INSPEC 



Ana. 



Non. 



F.I. 



TOTALS 



Ana. Non. 1 F.I. 



do 

(do, did, 
does , 
doing, 
done) 



2] 



42 



.33 



12 



.20 



24 



54 



62 



INDEFINITES 



CLASS SUMMARY 
PRELIMINAHY TEST SET 



TERM 


PSYCH. ABS. 




INSPEC 




TOTALS 




Ana» 


Non. 


F.I. 


Ana. 


Non. 


F.I. 


Ana. 


Non. 


F.I. 


all 


1 A 




• 35 


7 


17 


.29 


21 


43 


.33 


any 




/ 


• 22 




15 


0 


2 


22 


.08 


anyone 




1 


0 




1 


0 




2 


0 


anything 




1 


A 










1 


0 


each 








17 


24 


.41 


46 


52 


.47 


either 








2 


3 


.40 


22 


15 


.59 


enough 




1 


0 










1 


0 


every 




2 


0 




4 


0 




6 


0 


everything 


- 


















few 




6 


0 




4 


0 




10 


0 


fewer 


1 


9 


• 10 








1 


9 


.10 


least 




8 


0 




6 


0 




14 


0 


less 


10 


22 


• 31 




4 


0 


10 


26 


.28 


little 




4 


0 




1 


0 




5 


0 


many 


2 


10 


• 17 


1 


14 


.07 




24 


.11 


more 




49 


A A 

• 44 


11 


11 


,50 




60 


.46 


most 


2 


29 


• 06 


1 


8 


.11 




37 


.08 


much 




A 


• 4 J 


1 


2 


.33 


H 


D 


.40 


neither 


*> 
2 




1.00 


. 1 


1 


0 




1 


.66 


no 


3 


58 


.05 


2 ! 


13 


.13 


5 


71 


.06 


none 




1 


0 


1 1 

1 


1 


.50 


1 ! 


2 


.33 


nothina 




1 


0 








- 1 


1 


n 


several 




9 


0 




30 


0 


1 

i 


39 


0 


some 


1 


26 


.04 


1 


50 


.02 


2 ! 

j 


76 


.02 


someone 




















something 




3 










j 


3 


0 


TOTALS 


128 


317 


.29 


44 

e3 


209 


.17 


172 


526 


.25 



ERIC 



Adjective s Appendix A-8 

CLASS SUMMARY 
PRELIMINARY TEST SET 





PSYCH. ABS. 


INSPEC 




TOTALS 




TERM 












1 










Ana. 


Non. 


F.I. 


Ana. 


Non. 


F.I. 


Ana. 


Non. 


F.I. 


additional 






0 


1 


3 


.25 


1 


5 


.17 


another 


2 


3 


.40 


1 


3 


.25 


3 


6 


33 


both 


18 


36 


.33 


1 


25 


.04 


19 


.61 


.25 


else 










1 


0 


0 


1 


0 


equal 




5 


0 




1 


0 


0 


6 


0 


identical 


0 


2 


0 


0 


1 


0 


0 


3 


0 


TOTALS 


20 


48 


.29 


3 


34 


.08 


23 


82 


.22 











1 






t 

1 
• 














64' 













nppenaix A-if 



CLASS SUMMARY 



PRELIMINARY TEST SET 



TERM 



PSYCH. ABS. 



Ana. 



Non. 



F,I. 



INSPEC 



Ana. 



Non. F.I. 



Ana. 



TOTALS 



Non. 



F.I. 



here 

identical!^ 
similarly 
so 

such 
then 
there 
therin 
thus 

viceversa 



1 
3 
16 
1 



1 
8 
17 
14 
36 

10 



.50 
.27. 
.48 
.07 

0 

0 

1.00 



20 
1 
1 
1 



2 
1 

15 
27 
24 
24 



.33 

0 

0 

.42 
.04 
.04 
1.00 
0 



1 
3 
36 
2 
1 



4 

1 
1 
23 
44 
38 
60 
0 
13 



.20 
0 

.50 
.12 
.45 
.05 
.02 
1.00 
0 

1.00 



TOTALS 



22 



88 



.20 



24 



65 



96 



.20 



46 



184 



.20 



ERIC 
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RETRIEVAL EXPERIMENT 



Appendix b-i 



CLASS SUMMARY 





PSYCH. ABS. 




INSPEC 






TOTALS 


CLASS 






















/uid • 




1? T 


Ana. 


Non. 


F.I. 


Ana. 


Non. 


F.I. 


Central 


ZD / 


DD 




188 


29 


.87 


445 


84 


. 84 


Pronouns 


















Nominal 


]48 


]72 


.46 


169 


131 


.56 


317 


303 


.51 


Demons t rat 






• 










Relatives 


2 1 9 


168 


• •J t 


184 


82 


6Q 




OCA 


.62 


Nominal 


78 


7fi 


• D X 


49 


76 


39 






. 45 


Substitute 


! 














Pro- verb 


11 


20 


.35 


1 


17 


.06 


12 


37 


.24 


Indefinite 


> 112 


235 


.32 


35 


202 


.15 


147 


437 


.25 


Au jecuxves 


27 


51 


.35 


10 


40 


.20 


37 


91 


.29 


Adverbs 


25 


52 


.32 


30 


68 


.31 


55 


120 


.31 


S & Ss 


124 


25 


.83 








124 


25 


.83 


Definite 


277 


1303 


.17 


327 


1526 


.18 


604 


2829 


.18 


TOTALS 


1278 


2157 


.37 


993 


2171 


.31 


2271 

i 

i 

i 


4328 


.34 
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' Central • Pronouns 
(Personal, Possessive, Reflexive) 



Appendix B»2 



CLASS SUMMARY 



RETRIEVAL EXPERIMENT SET 



0) 



</) 

0) 
</) 
</) 

o 



0) 



a; 





PSYCH. 


ABS. 


INSPEC 




1 


TOTALS 


TERM 






















Ana. 


Non. 


F.I. 


Ana. 


Non. 


F.I. 


Ana. 


Non. 


F.I. 


he 


3 




1.00 






1.00 


14 




1*00 


him 


1 




1.00 






1.00 


1 




1.00 


she 


5 


- 


1.00 


1 


— 


1.00 


6 




1.00 


they 


28 


- 


i.oa 


17 


- 


1.00 


45 


- 


1.00 


them 






1.00 


Q 
O 




1.00 


22 




1.00 


his 






1.00 


/ 




1.00 


22 




1 A A 

1.00 


her 






1.00 






1.00 


13 




1.00 


its 


15 


- 


1.00 


42 


- 


1.00 


57 


- 


1.00 


their 


123 


— 


1.00 


39 




1 on 


Xu ^ 




1.00 


herself 


1 




1.00 






1.00 


1 




1.00 


himself 






1.00 






1.00 






1.00 


itself 


2 




1.00 


3 




1.00 


5 




1.00 


themselves 


9 




1.00 


3 




1.00 


12 




1.00 


sub-total 


229 




1.00 


131 




1.00 


360 




1.00 


it 


28 


55 


.34 


57 


29 


.66 


85 


84 


.61 


TOTALS 


257 


55 


.82 


1 

188 1 

1 

j 

i 
1 

1 
1 


29 


. 87 

1 
1 

1 


1 

445 

t 

1 

! 
1 
I 


84 


.84 
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i 
1 

1 

( 
1 

t 

1 

I 

1 

1 









ERiCl 



CLASS SUMMARY 



RETRIEVAL EXPERIMENT SET 



PSYCH. ABS. 



TERM 

this 
these 
those 
that 



TOTALS 



Ana. 

56 
52 
25 
15 



148 



Non, 

7 
1 
1 

163 



172 



F.I. 

.88 
.98 
.96 
.08 



.46 



Ana. 

88 
64 
6 
11 



169 



INSPE C 

Non. 

54 

2 

6 
69 



131 



F.I. 

.62 
.97 
.50 
.13 



.56 



Ana. 

144 

116 
31 
26 



317 



TOTAL S 

Non. 

61 
3 
7 

232 



303 



F.I. 

.70 
.97 
.82 
.10 



.51 



69 



CLASS SUMMARY 
RETRIEVAL EXPERIMENT SET 



PSYCH. ABS. 



TERM 



Ana. 



who 

whom 

whose 

which 

where 

that 



TOTALS 



63 
3 
6 

68 
6 

73 



219 



Non. 



4 
62 
162 



F.I. 



168 



1.00 
1.00 
1.00 
.94 
.75 
.31 



.5' 



Ana. 



INSPEC 



Non. 



4 

123 
4 
48 



184 



1 
8 
69 



82 



70 



F.I. 



Ana. 



TOTALS 



Non. 



.55 

1.00 
.99 
.33 
.41 



.69 



68 
3 

10 
191 

10 
121 



5 
10 
231 



403 250 



CLASS SUMMARY 



RETRIEVAL EXPERIMENT SET 





PSYCH. 


ABS. 




INSPEC 




TOTALS 


TERM 












1 PI 










Ana. 


Non . 


F.I. 


Ana. 


Non. 


F.I. 


Ana. 


Non. 


F.I. 


above 


1 


1 


.50 


2 




1. 00 


3 


1 


.75 


former 




















last 


1 




1.00 


1 


3 




2 






latter 


2 




1.00. 


3 




1.00 


5 




1.00 


one, ones 


10 


15 


.40 


10 


25 


.29 


20 


40 


.30 




43 


9 


. 0 J 


1 c 
i.J 


4 


.79 


do 


13 


.82 


others 


3 


]2 


.20 


2 




1.00 


5 


12 


.29 


same 


2 


6 

■j 


.25 


6 


1 


.86 


8 


7 


.53 


Sub-Totals 


62 


i 43 


.59 


39 


33 


.54 


101 


76 


.57 


first, 1st 


7 


13 


.35 


5 


27 


«16 


12 


40 


.23 


second, 2ni 


i 6 




.75 


4 


13 


• 24 


10 


15 


• 40 


third/ 3rd 


2 


r 
D 


.29 


1 


2 


. 33 


3 


7 


. 30 


fourth /4th 


1 


4 


.20 








1 


4 


.20 


fifth /5th 




2 


0 




1 


0 




3 


0 


sixth /6th 




5 


0 


- 






- 


5 


0 


seventh , 7t\i - 


1 


0 










1 


0 


eighth, 8th 




1 


0 


- 






- 


1 


0 


ninth, 9th j 




















tenth^lOthj 








- i 






i 
1 






Sub-t6tals 


16 


33 


.33 


10 


43 


.19 


1 

26 1 

i 


76 


.25 


TOTALS 


78 


76 

1 


.51 


j 

49 i 


76 


.39 

I 


127 ' 

1 

! 


152 


.45 






1 
1 

1 
i 
i 


■ 


71 




1 
i 

k 

1 

1 


1 
1 







CLASS SUMMARY 
RETRIEVAL EXPERIMENT SET 



TERM 



PSYCH. ABS. 



Ana. 



Non. 



F.I. 



INSPEC 



Non, 



F.I. 



Ana. 



TOTALS 



Non. 



F.I, 



Do 

(dO/ did, 
does , doing 
done) 



11 



20 



35 



17 



72 



.06 



12 



37 



.24 



'ERiC' 
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CLASS SUMMARY 
RETRIEVAL EXPERIMENT SET 



TERM 


PSYCH. ABS. 




INSPEC 


TOTALS 
























Ana. 


Non. 


F.I. 


Ana. 




IT T 


Ana . 


Non . 


F.I. 


all 


IZ 


17 


• 41 


2 


28 


.06 


14 


45 


. 24 


any 


X 


n 
9 


.10 


2 


13 


.13 


3 


22 


• X2 


anyone 












— 


— 


— 




anything 






_ » 






— 


— 


- 




each 


28 


14 


.66 


17 


14 


.55 


45 


28 


• 62 


either 


Id 


2 


. 88 


6 




1.00 


21 


2 


• !^X 


enough 










3 


0 




3 


ft 
U 


every 




2 


0 




6 


0 




8 


U 


everything 






U 










1 


A 

0 


few 




c 


f\ 

V 




4 


0 




9 


n 
u 


fewer 


X 


c 
D 


. 17 








1 


5 


• X / 


least 






0 




4 


0 




1 

7 


A 
U 


less 


A 
H 


1 n 
iv 


.28 


1 


2 


.33 


5 


12 




little 






0 




2 


0 




11 


A 
U 


many 


1 
X 




• 10 


1 


33 


.03 1 


2 


42 




more 


A t\ 

40 


56 


.42 


3 


30 


• 09 I 


43 


86 


• 


most 


3 


20 


.13 




7 


^ i 

0 1 


1 

3 ! 


27 


• xu 


much 






0 


2 


8 


.20 


2 


11 


• xd 


neither 


X 


c 
D 


. 16 








1 


5 


• xo 


no 


2 


29 


.06 




4 


0 

j 


1 

2 : 


33 


• uo 


none 


1 


1 


.50 










1 


C A 

.50 


nothing 




1 


0 


- 




1 


_ i 


1 


0 


several 


1 


11 


.08 


1 


20 


.05 


2 : 


31 


.06 


some 


2 


22 


.08 


_ I 

1 


24 


0 i 


2 : 


46 


.04 


someone 




1 


0 


- 1 




r 

i 




1 


0 


some uning 




1" 








1 


- j 






TOTALS 


112 


235 


.32 


35 


202 


. 15 

1 


147 ' 

1 

f 

1 




.25 


■ ■ 1 




1 

i 




7- 




i 

i 

1 









CLASS SUMMARY 



RETRIEVAL EXPERIMENT SET 



TERM 



additional 

another 

both 

else 

equal 

identical 



PSYCH. ABS 



Ana. 



2 
25 



Non. 



3 
7 
32 
1 
2 
6 



F.I. 



0 

.22 
.44 

0 
0 
0 



INSPEC 



Ana. 



6 
3 



Non. 



F.I, 



4 
2 
32 

1 
1 



0 

.75 
.09 

0 

.50 



TOTALS 



Ana. 



8 
28 



Non. 



7 
9 
64 
1 

3 
7 



F.I. 



0 

.47 
.30 
0 
0 

.12 



TOTALS 



27 



51 



35 



10 



1{ 



40 



,20 



37 91 

I 

I 



.29 



'ERLC' 
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CLASS SUMMARY 



RETRIEVAL EXPERIMENT SET 



TERM 



PSYCH. ABS. 



Ana. 



here 

identical!^ 
similarly 
so 

such 
then 
there 
therein 
thus : 
viceversa ! 

L 



1 

3 
21 



Non. 



3 
8 
6 
29 



F.I. 



1.00 
.50 
.72 
0 
0 



TOTALS 



25 



52 



32 



Ana. 



INSPEC 



Non. 



27 
1 



19 
19 
9 
12 



30 



68 



P.I, 



0 

.59 
.10 

0 

1.00 
0 

1.00 



Ana. 



TOTALS 



Non. 



1 
3 
48 

1 



22 
27 
15 
41 

11 



3.1 



55 i 120 



75 



Paqe 78 



P.I. 



1.00 
.12 
.64 
.06 

0 

1.00 
0 

1.00 



.31 




CLASS SUMMARY 
RETRIEVAL EXPERIMENT SET 



TERM 



S and Ss 



PSYCH. ABS. 



Ana. 



124 



Non. 



25 



F.I. 



.83 



INSPEC 



Ana. 



76 



Non. 



F.I. 



TOTALS 



Ana. 



124 



Non. 



25 



F.I. 



.83 




■ERiC: 



Page 79 



CLASS SUMMARY 
RETRIEVAL EXPERIMENT SET 



PSYCH. JiBS. 



TERM 



the 



Ana. 



277 



Non. 



1303 



F.I. 



.17 



INSPEC 



Ana. 



327 



Non. 



1526 



F.I. 



.18 



77 



Ana. 



604 



TOTALS 



Non. 



2829 



F.I. 
.18 



'eric 



Paqe 80 



APPENDIX C 
Linguistic Analysis 




APPENDIX C 



This appendix contains rasulta of our datsiled linguistic 
analysis of all tha potential ly anaphoric tarms uhich uara 
obsarvad to function anaphorioal ly in tha sampla of 600 
abstracts. This analysis attaaptad to dalinaata the epecific lex- 
ical environsents which could bs ussd to reliably predict whether 
a tars was anaphoric or not in specific instancee. 

These ruls-oriented analyees thsn served as the basis for the 
rule-sets tested by independent Judgee to determine whether the 
P.A./F.A. distinction was adequately captured in the rules. The 
high succsss rate of that teeting (eee Appendix 0) indicates that 
theee algorithmic type rulest once captured in formalized codet 
aay be ueeful in enabling a system to determine automatically 
whether a P. A. is an F. A. 



Contente of Appendix C 
Cen t ra 1 P ronouns 

Mt» 

Kos i na 1 Demons t rat i vee 
Relative Pronouns 
Nominal Substitutes 
♦One» 
'Same* 
•Other* 
•Othere* 
Ordinals 
Pro-verb 'do* 
Indef ini tee 
Univereals 

♦Each» 

♦An» 
Hultale 

* Many* 

•Here* 

♦nost» 

•Much' 
Paucala 

'Leee' 
Assert i vee 

'Some-' group 
Non- Aeee r t i ves 

'Any' 

'Either' 
Negatives 

'No' 

Residua! Adjectives 
Adverbs 

•So' 
'Such' 
Subjecte 
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central Pronouns 



Quirk l( GrMnbaum sub-divids thm Mjor claeo of 'Central Pro- 
nouns* into thsss thrss minor classssx 1) Personal pronouns; 2) 
Posssssivs pronounsf andl 3) Rsflsxivs pronouns. The individual 
nesbers of these claeses are: 



Persons! I Jj ne» jis., liSL* you* lift, lijji, iJift, iiajp,tl3 lt,t23 thev , 
thes « 

Possessive: myt ninet oure, your , youre, 22jjBL» her , herst 

ilAt ilULix* thei rs« 

Rsflsxivst syself* ourselvest yourself, youraelvest himeel f t 
hereelff LisslL* themeelves. 



Only those pronouns which are underlined were observed in the 
subset of 600 abstracts. 

Ths anaphoric use of these three typee of pronouns can be pre- 
dicted by the person distinction which pronouns demonstrate (1st 
persont 2nd person, 3rd psrson). Of the 325 occurrences of the 
Csntrsl Pronouns, all 1st and 2nd person pronoune (15 occurrenc- 
ee)f whether pereonal, possessive, or reflexive were non- 
anaphoriCf while all 3rd person pronouns (310 occurrences) were 
anaphoric* 

The non-anaphoric uses are deictic references to either the 
author(e) of the abstract: 

(1) Our research complements the EPA guidelines... 

(2) The system that we are developing* • • 

or to rather indeterminate, unspecified individuals: 

(3) Discovering your radiant eelf. (title) 

(4) Three paradoxes ars considered: (a) We hurt and are hurt 
by thoee we love... 

(5) Exploree the idea that Gestalt concepts apply to our 
physical as well as our msntal bsing. 



It is unnscessary, therefore, to develop rules to determine 
uhether in a particular instance a csntral pronoun is anaphoric 
or not. Automatic matching against lists of pronouns tagged for 
person-distinction should suffice to locate anaphoric references. 
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Ill B«lon00 to both Parsonal and PosasMive classes 
121 ^It* is handlsd in a saparata analysis. 
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IT 



Froa InforMtion found in standard graaaar aourcaa* i it 
appaars that 'It* haa four poaaibla uaaa* only tuo of which hava 
baan obaarvad axtanaivaly in tha aaaplaa of abatracta. Tha othar 
two umaa had only 1 occurranca aach. All four uill* houavar* ba 
datailad hara ainca it ia nacaaaary to uaad out all non-anaphoric 
uaaa. Tha firat thraa uaaa ara non-anaphoric and thay will ba 
praaantad in ordar of thair aaaa of diatinguiahability froa tha 
othar caaaa. The anaphoric uaa will ba praaantad lart ainca it 
doaa not occur in aa p radio tab la a syntactic anvironaant aa do 
tha nonanaphoric uaaa. 



Capty/Props Mt* aay ba uaad to rafar to tha rathar indatarai- 
nata notion of tha ganaral atata of affaira. Fraquantly this 
uaa ia to do with tha waathar or tha tiaa. 

(2) It ia raining out. 

(3) It ia nina-thirty. 



Tha naxt two uaaa appaar to ba apacial caaaa of tha nor a gan- 
aral notion of cataphoric uaa of *it*. In both uaaa tha rafarant 
for which *it* ia aubati tuting* foUowa Mt* in tha taxt. 

Anticipatory: Mt' appaara aa tha raault of raarrangaaant of 
taraa froa tha uaual word ordar by tha aovament rula 

known in tranaf oraat ional graaaar aa axtrapoait ion. Thia 
involvaa aovament of a clauaal aubjact froa tha original ayn- 
tactic atructura ofs 



clauaal aubjact pred 



to a poaition toward tha and of tha aantanca. Tha poatponad 
alaaant'a poaition ia fillad by tha anticipatory pronoun Mt*. 
Tha raaulting syntactic atructura ia: 

*it* ^ prad ^ clauaal aubjact 

Obaarvation of tha abstracts ravaals that tha pradicata in this 
typa construction appaars to ba aithars 

1. Of tha class of cogni tiva/anotiva varba of thinking* 
knowing* faaling* atc.» followed by * that* and an inda- 
pandant clauaal 

(4a} It is aaphaaizad that tha avidanca waa 
obtained froa noraal children reared in their 
natural hoaea by their biological parenta. 



1 Quirk* Greenbauaf Leach Ic Svartvik. A Graaaar of Contaaporarv 
Enol igh » Longnan Group* 1960. 



82 



ge 85 



. ' ' i 



»£or». SoK« coMon construction* mtmi Mt it po»«ibl« 
for' s Mt was difficult for' I Mt is unrnlistic to'. 

(5«) It is crucial for thsrapists to fssl frss to 
discuss usss and abusas of this aonay with 
patiants* 

In all inatancas of axtrapositiont ona can aaaily raarranga tha 
sentanca alaaants to raturn to noraal S-V-0 ordar by substitut- 
ing tha clauaal subjact for Mt'» 

(4b) That tha avidanca was obtainad froa noraal 
childran raarad in thair natural hoaas by thair 
biological parants is aaphasizad* 

(5b) For tharapists to faal fraa to diacuss usas 
and abuaas of this aonay with patianta is cru- 
cial* 

Claft santancas Mt' is usad in conatructiona of thia typa to 
parait' focal proainanca to ba givan to a particular itaa in tha 
santanca* Santanca alasaants ara raarrangad froa normal ordar 

to: 

'It' ^ fora of 'to ba' focus alaaant ral* clausa 

(6) It uas tha uaathar that cauaed tha picnic's 
cancal lation. 

Claft santancas can ba di f farantiatad froa anticipatory con- 
structions by tha fact that tha clauaa postponad in anticipato- 
ry usaga is an indapandant clauaa in uhich tha subordinating 
conjunction does not fill a syntactic slot. On tha othar handt 
in claft santancas tha haad of tha ralativa clausa fills a syn- 
tactic rola in tha clausa* 

Anaphorict 'It' parforas as an anaphoric itaa whan 'it' is in 
its rolo as a parsonal pronount i^ a. • it sarvas as an abbravi- 
atad rafaranca to a aora fully axplicatad antacadant. Houavart 
'it* diffars froa all othar paraonal pronouns in that 'it' has 
tha capability of axtandad rafaranca. 'It' aay raplaca a uhola 
clausa or santanca or 'it' aay sisply rafar to a singla ucrd. 
Alsof in anaphoric usagat 'it' aay torn ralatad to its antacadant 
aithar by 'identity of rafaranca' or 'identity of apecifica- 
tion'. In identity of rafarancat 'it' refers to the exact saaa 
entity as the antecedent. 

(7) Feedback has an iapact on the atrangth of 
beliefa to which it ia targeted. 

Vhereas* in identity of apecif icationt Mt' refers to a aaparata 
entity but one that ie specified in saaa aannar aa its antece- 
dent. 
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Bill ordered it too. 



Norn 1 na 1 Demonst rat i ves 



Damonatrativa rafarance ia aauantially a form of verbal point- 
ing* Thara ara 4 nominal damonatrat ivaa: *thia*» * theaa* t 
'thoaa'9 and »that». Tha nominal damonatrat ivaa »thia»f 'thaaa't 
•nd *thoaa' function only aa rafarantial itama. Thay hava no oth- 
•r uaa. 'That* haa four aanaaa aaaociatad uith itt and will ba 
traatad aaparately. 



THIStTHESEtTHOSE 

Whan any of thaaa 3 tarma ara ar^ountarad in tha textt what 
muat ba datarminedt thoraforat ias 

l» Vhathar tha rafaranca ia aituational (axophor ic/daict ic) or 
taxtual (andophoric) • 

2. If andophoriCf uhathar tha rafaranca ia backward in text 
(anaphoric) or forward in text (cataphoric). 

I 



{i&„<K^ie>Ka\ /ci e»c+rc ) C+ to aO 



I 

In Baking tha firat determination, tha fact that abatracte are 
quite ael f-contained and non-aituational ly dependent predicta 
that the endophoric uaa ia common and tha exophoric quite uncoa- 
mon. Thia haa been obeerved to be the caae* Therefore* it ia more 
efficient to proceed by deteraining tha contextual cluea (lexi- 
cal* not syntactic) that indicate exophoric uae* rather than 
cluea to andophoric uaa. 

There exiat two general caaea of exophoric uae of nominal 
demonatrativaa aa exhibited in abatracta. The firat of thaaa ia 
deictic reference to eithert i) the document of which the 
mbatraet ia a part on 2) tha tiae at which tha document waa 
written. *Thia* ia the ueual noainal deaonatrat iva choaan for 
auch use and typical phraaea ares » in thie paper'. »at this writ- 
ing** or 'this report'. 

E.g. **The diacuaeion of aexual behavior in thia paper ia 
confined to heteroaaxual activi t iaa. ** 
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tivM •XMplifiM the larcar phsnomsnon of rsfsrlng to indetermi- 
nmtm rafarsnts uhich mrm prMunsd to sxist but which mrm not 
cp«cifi«d« PhrasM conpossd of double pronouns «uch as * thoaa 
uho* f *anyona uho*f or *that which* ara conmon laxical indicators 
uaed to rafar to aoaaona oraoaathing without actually denoting 
anyone o rany th i ng • 

E.g. **Paopla who buy aocial acience should reaember that 
data can aaaily be aiaconatrued or misrepresented by 
thoss who wish to prove their particular argument* for 
any of a number of reaaona.** 

The aacond deterainatiori ia whether the endophoric reference 
is snaphoric or cataphoric. In abatracta* cataphoric noun phraa- 
ss are uaed to introduce a liat* and are usually followed by a 
colon.C 1 ] 

E.g. ""The experiment tested these three approaches:** 

Having sliminated the non-functioning P.A.^Sf the F.A.*s may 
be classified. *Thie** *thess' and * those' function anaphoricaly 
either as: 

0 Dssonstrative adjsctivs 
^ Dssonstrative pronoun 

Cleseify es demonstrative adjsctivs if the term is followed by a 
noun or an adjective. Otherwiss* classify as demonstrative pro- 
noun. 



THAT 

*That* has four ssnsea aaaociated with itr Three of theae are 
referential uaee and the reference is anaphoric for each uee. 

0 Deaonatrative adjective 

0 Deaonatrative pronoun 

« Relative pronoun 

Non-anaphorical ly* *that' functione eat 
• Subordinating conjunction 

To dataraina in a particular instance whether 'that* ia an F.A.* 
the one non~*anaphoric uae will be tested fort and all auch uaee 
exoludcid froa further analyaia. 
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inmx,' 9 xn i ls roi« as a suoorainat ing conjunction* occurs in 
two contaxts and appaara to torn acting m a laxical colon in both. 
In ona contaxt» tha rola of 'that* as a subordinating conjunction 
is rscognizabla by theaa tuo facts. 

1* *That* folloua cogni tiva/aaotiva varbs of: 

0 knouing 

0 thinking 

o bal iaving 

o faaring 

0 saying 

o raaaabaring 

o parcaiving 

or thair noainal isat ion: 

o assuaption 

o suggastion 

o hypothaais 

o sxplanation 

o suggaation 

2, Tha clausa introducad by 'that* contains no empty syntactic 
slot* i. a. tha clauaa is complatSt it consiats of subject- 
varb-objactf in any order* 

E.g. **It uaa datarainad that a vary definite advantage ia 
achieved uhen the airflow is reversed periodically.** 

One eeesingly troublesoae construction* *that is* * is actually 
an sUipssd variant of the phrase *that is to say* and aervea aa 
en indicator of a subsequent phrass of apposition. The ellipsed 
verb *say* bslongs to ths claas of cognitive verbs which indicate 
uaa of •that* as a subordinating conjunction. Therefore, the 
construction 'that is* will be clasrifisd as such. 

E.g. **ln the first* the concern is to construct a reaia- 
tivity structure whose responses are acceptably cloae to 
the observations* that is* ths ssasured aaplitudes and/or 
phrases. ** 

In the sscopd context* 'that* ia one cosponent of a coapound 
subordinating conjunction, and ia recongnizable by two facts: 
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A. ' inmx.' xm ins zinai •ismeni. in constructions oz tlis zoilou- 
ing typst 



0 


but that 


0 


in that 




such that 


0 


so that 


o 


in ordsr that 



2. Again in this contsxti ths clauas introduced by * that* con- 
tains no sapty syntactic slot. 

E.g. **Skinnsr*s eoncspt of contingencies of rein- 
forceaent aay be a crucial one for understanding the 
relationship between the arts and the sciences in 
that each involves processes and products of human 
behavior. 

Hfiving excluded non-anaphoric occurrences of ' that* » the 
remaining instances aay be classitied. 

0 Classify as relative pronoun if 'that* introduces a clause 
that is not coapletet i.e. contains a syntactically empty 
slot. 

E*g. ^Phillips developed a system that diagnosed 
human ill ness . ** 

^ Classify as deaonstrative adjective if followed by noun or 
adjective. 

E.g. **Selected components of that f rameucrk are 
smpirically tested.** 

^ Otherwisst classify as d<jmonstrative pronoun. 

E.g. *"Ths perforsance of the model is compared to 
that of the physicians.** 
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11 J Basttd on three cetaphoric inntances in sample of 600 
abstracts. 
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n«i«i. 1 vv rronouns 

Rttlativs pronouns introduce ralativs clauses postaodi fying 
noainal hsada* and havs anaphoric refsrance to the antecsdsnt 
noun phrass which is postaodi fisd by tha entire relative clauss. 
This class ot pronouns consists of the following terse: who* 
whOB» whosSf which* and that.Cl] All of thess terms were observed 
in the subsst of 600 abstracts. 

All occur JT'sncee of *who*» *whoa'» and * whose* wars anaphoric 
while 4 of the 195 ocurrences of * which* were nonanaphoric. 
Therefore* only rules for deteraining anaphoric vs. nonanaphoric 
uss of * which* were developed. 



Anaphoric Uss 

Ths anaphoric use of 'which' occurs in three different syntec- 
tic environssnts. 

1. *Vhich* aay follow iaaediately the noainal head it postao* 
difies. 

(1) Performance is coapared with the traditional algor- 
ithm which employs only snapping. 

2. * Which* aey be immediately preceded by and function as 
objsct of a prspositicn. 

(2) The process is aodslled by a hyperbolic systea in 

the inflows act both as distributed and as boundary 
controls. 

3. *Vhich* may follow a vsrb in tha passive voice* which sepa- 
ratss ths ralative pronoun froa the nominal head it postao- 
di flees. This usags can be deterained by the fact that these 
passive verb phrasee csn bs aovsd to the end of the rela- 
tive clauee without altering the aaaning of the ssntancs or 
daaaging its graaaatical ity. 

(3) An algoritha is prssented which maps patterns froa a 
high-dimensional specs to a plans. 



Non-anaphoric Use 



0 *Vhich* in ite nonanaphoric usage acts as an indefinite det- 
srsinsr of the noun phrase which follows it. The typical 
syntactic snvlronasnt for this usags is: 

verb ♦ * which* ♦ noun phrase 

The verb phrass is usually activs snd can in no way be moved 
without daaaging the graaaatical ity and sense of the sentence. 
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i4) The atudy uill atteapt to determine which nethod of 
enelyeie will be noet coat-effective. 





11 J mat amm treatea in tne analyaia of 
and uill not ba raconaidarad hara. 



nominal defflonatrativaat 
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Nominal Substitutes 



Th« mmt of terms considered as nominal substitutes was com- 
pleted with the following summary results: 





P. 


A. 


Ins. 




Total 






Ana. 


Non. 


Ana. 


Non. 


Ana. 


Non. 


F. I . 


above 


0 


3 


1 


0 


1 


3 


.25 


former 


o 


0 


2 


0 


2 


0 


1.00 


last 


0 


2 


0 


2 


0 


4 


0 


latter 


3 


0 


4 


0 


7 


0 


1.00 


one 


7 


17 


8 


30 


15 


47 


.24 


other 


33 


8 


27 


4 


60 


12 


.83 


others 


4 


6 


6 


0 


10 


6 


.62 


same 


13 


15 


4 


11 


17 


26 


.40 


Total 


60 


51 


52 


47 


112 


98 


.53 



Obviously* the terms 'former' and 'latter* which have an F.I. 
of 1.00 will not be teeted? nor will 'last' which had an F. I. of 
0; nor will 'above' which had only 1 anaphoric use in the set of 
600 abstracts. 

The remaining four terms - 'one'* 'other'* 'others' and 'same' 
have separate rule sets for each term. , 
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ONE 



Th« t«rB 'one' (including th« forms 'ones' and 'one'sM has 
three eajor senses associated with it. 'One' may be used as: n a 
numsrall 2) a nominal substitutst orl 3) an indefinite pronoun. 
To determine »hich of these senses is intended in a piece of 
textf it is first nscessary to undsrstand the detailed structure 
of a nominal group, (a.k.a. noun phrase) 

NOHiNk I. Gk^^^ 



(1) the six red onions on the table 

(2) the difficult ones 

(3) ons method 

(4) one current technique 

(5) that smoking gives one cancer 

The slot in ahich tne term 'one' occurs within the nominal group 
will determine which use of 'one' is intended. 
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HOffiTTsl fV^)it^tute 

If 'one' functions as head (d) of a nominal group premodified 
by either m determiner (m)f e. g. definite article or nominal 
demonstrativSf or an adjsctive (c) or botht as in (2)f the term 
is bsing ussd anaphorical ly as a nominal substitute. The syntac- 
tic snvironmsnt would be: 

premodif ier (s) ^ 'one' 

(6) The store had no gold bracelets; Just silver ones. 

In its uss as an indefinite pronoun* 'one' is non-anaphoric in 
that thsre is no preeupposi tion of a sore specified antecedent to 
which *one' is referring. Its msaning is that of an indetermi- 
nate? generic person who cannot be defined any more specifically 
within the text. 

(7) One neve^r knows what might happen. 
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In terms ox the nominal group structurst the indefinite pro- 
noun *one* has been observed in this data set to occur as the 
unmodified head <d)t as in (5). The form *one*s* is found only 
in this usage. 



The most frequent uee of *one* is as a cardinal number. In 
soss instancss this senss of *ons* i^ non-anaphoric. In otherst 
whvn its uss is combined with the linguistic technique of el lip- 
siSf it is anaphoric. Ths sasiest non-anaphoric structure to rec- 
ognize is the hyphenated combination. 

(8) One-sided ssquential teste for the mean of an 
exponential distribution are proposed. 

The remaining occurrences of *one* is its use as a numeral can 
be detected by again referring to tha nominal group structure. In 
its un-hypenated numeral uses* ^one* functione as a premodifier 
in a nominal group* as in <3) and <4). The structural environ- 
ment would be: 

*one* ^ {adjective) {head) {prepositional phrase) 

In other words* when ussd as a numeral* 'one* is not preceded 
by another premodifier* but aust bs succeeded by at least one and 
possibly sven all of the following: 

adjective - c 
head noun - d 
prepoeitional phrase - e 

(9) A control function is propo&ed for one possible 
systsm conf ijiuration. b ^ 

(10) The conjecture is shown to be true for one level 
of Mi ext* s tatement^ . b A 

c 

(11) The evaluation of textbooks using one A^the 
standard Readability formulas, is a lengthy ^ask. 

The only exception to this rule as observed ;n the 600 abstracts* 
w-as the restrictive adjective •only* which preceded »one» twice 
in the data set* although *one* was bsing used as a numeral. 

To then determine whether this usage of *one* is anaphoric or 
not* ths prior tsxt must bs scannsd for an earlier occurrence of 
the head noun whsich *bne* is modifying. If that head noun is 
specified in greater detail in a prior usags* then *one* is to be 
considered anaphoric* sines its usags sstablishes an acceptable 
environaent for some premodifiers to bs sllipsed. 

(12a) This is illustrated by a detailed examination 
of two simple microprocessor-based gaging systems. 
0ns system measurss location. 
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vn«n tn« snaphoric ^onm^ im rMOlv«d in this usage* the ellipsed 
prModifierv are re-ineertad. 



(12b} This is illustrated by a detailed examination 
of two simple microprocessor-based (paging systems. 
One sisple microprocessor-based gauno system measures 
Ideation. ^ a 

'One* has been observed in this data set to be anaphoric only in 
the environsent: 

•one' + head noun 

although the inverse of this is not true. That is* all instances 
of *une' in this environment are not anaphoric. 
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Page 86 
Appendix C-19 

'SAME* 

*SaMtt* occurred in thm 600 abstracts a total of 43 tiaas with 
17 of thasa occurrancaa baing anaphoric for an F. I. of .AO. 3 
of tha 4 syntactic anvironaants in which *saaa* uaa obaarvad ara 
always non-anaphoric* whila tha statua of *saaa* in tho 4th anvi- 
ronaant dapands on prior taxt. 

Mon- Anapho r i c 



1. 

*tha* **- *saaa* **- praposition 
»or» 

•tha» ♦ *saaa* ♦ . 

1. Her rasponsas reaainad tha sama throughout tha interroga- 
tion. 



2. 

*saaa* **- noun 

2. Tha atudants interviewed were a very hoacgenoue group 
saae likest saae dislikes. 




3. 



*the* *»- 'saae* ^ adjective ^ noun 

3. The aajority of reapondents indicated an interest in the 
aaae leieure-tiae activities. 



Dependent on Text 

Vhen the following syntax is encountered: 

•the' ♦ *saae* noun 

•saae* is non-anaphoric if the noun it pre*-aodif iast or that 
noun's synonya* was not used earlier in text in a acre fully 
aaplified reference. 

On tha other hand* 'saae' is baing used anaphorical ly if the 
noun it pre-aodif ies was specified earlier in text in fuller 
detail* The earlier specification aay be in in the fora of 1 or 
sore pre-aodif iars of the noun* which are ellipaed when 'saae' is 
used in the current reference. 

4. Expert searchers used the full -text approach. Novice 




searchers used the saae approach. 



ERIC 
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Page 87 
Appendix C-2C 

Ort the noun ummd uith *saa«* mmy bm m rather o^neral tern 
which vmm explicated earlier in acre detail by either a prepoei** 




tional phrase: 



5. Freehaen were aost concerned uith the problea of having 
to choose a aajor. Soae sophoaores reaained disturbed by the 
saae problem. 

Ort the aarlier reference aay have been a detailed explanation 
not even contaHnino the saae general tare or its synonya. 

6. 15 Subjects were exposed to the stinulus for 4 ainutes 
while 15 Subjccte uere exposed to the control condition for 
the saae interval* 
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Page 88 
Appendix C-21 

» Other » 




•Other' occurred in the 600 ebetrects a total of 72 times with 
60 of these occurrences being anaphoric for an F. I. of .83. 

1. The basic uee of »other» is to sake some kind of a compari- 
sont but in most instances the comparison is not as fully 
spslled out as the underlying 8ean7ng intends. The sost 
typical comparisons are of the form: 

(la) This beer is sold in the U. S. and 14 other 
countries. 



which would be resolved by moving »other» to a position following 
the nount adding the explicit comparative term »than»t and that 
which is being compared: 

(lb) This beer is sold in the U. S. and 14 coun- 
tries other than the U. S. . 



The typical syntax for this use would be: 



•other' + (adjective) + noun 



end tho use is anaphoric in almost all instances except those feu 
where there is no information given as to what the »othBr» entity 
is being compared to: 

(2) This beer is sold in 14 other countries. 



There are 3 additional possible syntactic environments for the 
anaphoric use of 'other'. 

2. 'Other' may combine with 'each' in a reciprocal reference: 

'each' + 'other' 



3. 'Other' may be used as a pronominal in the following syn- 
tax: ^ 

'the' + 'other' (not followed by a noun) 

(3a) There are two transformational grammar 
approaches. The first builds on Chomsky's work 
end the other follows Postal's model. 

which would be resolvsd as: 

(3b) There are two transformational grammar 
approaches. The first transformational grammar 
approach builds on Chomsky's work and the other 
transformational grammar approach f ol lows Post- 
al 's model. 
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Page 89 
Appendix C-22 



4. Vh^n the explicit comparatives *than* or *uhile* precede 
the noun phrase containing * other* t again a comparison is 
being sadet but one uhich uould be resolved differently 
than is the case in #1. 

•than* /•while* + 'the* + •other* + noun phrase 

(4a) Groups of catst dogst and rabbits were 
exposed to the same stimulus. Dogs performed 
better than the other groups. 

uhich uould be resolved as: 

(4b) Groups of catst dogst and rabbits uere 
exposed to the same stimulus. Dogs performed 
better than cats and rabbits. 



As uas pointed out in #1> there is one syntax in uhich * other* 
say be either anaphoric or non-anaphoric* but there is only 1 
soley non-anaphoric syntax for *other*s 

•other* + •than* 

(5) Universities other than S. U. have an over- 
emphasis on sports. 
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Page 90 
Appendix C-23 

•OTHERS' 

* Others* appeared in thm 600 abstracts a total of 16 tinaa 
uith 10 of thaae uaaa baing anaphoric for an F. I. of •62. 

Mon-Anapho r i c 

Vhan uaad non-anaphorical lyt *othara* rafara to indafinita 
individuals uhose spacific idantity im of no concarn. Tha non* 
anaphoric usa of 'othars* alsost always follows praposi tionss 

(1) Concarn for othars Is not highly valusd in 
this sociaty. 



Vhan usad anaphorical lyt * othars* sarvas as a pronominal sub* 
stituts for individuals or itass rafarrad to aarliar; parhaps 
svan anuaaratad and has tha saaning of *sora lika tha abova* • 

(2) Ss axhibitad tha dafansa aachanisas of deni- 
al « projection and others. 

In this anaphoric usst 'others* s:lther follows 'and* or functions 
as subject or direct object of thm sentence. 




• 
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Page 91 
Appendix C-24 

ORDINALS 

Thm ordinailat which ar« groupttd uith thm noninal substitutM 
in this studyt trom 'first* to *tsnth* umrm obssrvsd in ths 600 
abstracts as follows with an ovsrall F. !• of .25. 

Psych Abs Inspsc 

Ana* Non. Ana. Non. 

first - 5 8 12 

sscond - 2 3 7 

third - 3 1 

fourth - I • 2 

fifth - 1 * 1 

sixth • • - 1 

savanth - - - * 

sighth - - * 

ninth - • - 1 

tonth ^ - ^ 

Totals - 12 12 24 



NoT|-AT|fpftoric Ugf 



1* Hyphsnetsd tsras in which ons tsrs is an ordinal ars always 
nonanaphoric* Soas cosaon usas of this typa ara: * sscond- 
gradsrs't * f irst--ordsr calculus*t *ons-sixth*. It doss hsp- 
psn infrsqusntly that ths hyphsn is oaittsd* but ths notion 
intsndsd by ths two tsras is obviously that of a known 
hyphsnatsd tsrs. 

2. Titlss of asstingst bookst stc. frsqusntly uss ordinals 
nonsnaphoricallyt m. g. 'Sacond Edition* t ^Eighth Annual 
Hasting*. Ordinals srs slso ussd in Isss foraal titlss such 
ss * fifth gsnsration oosputsrs* • 

3. Ths ordinal 'first* functions nonsnaphorically as an advsrb 
with tha asaning of **bsfors anothar in tiss or spacs or 
sction**. Typicsl syntsx for such a uss ist 

auxiliary vsrb * first* sain vsrb 

(1) Sub j sets wars first taggad and than ralsased to 
ths snvironasnt. 

or 

*at first* 

(2) At firstt both tschniquss sppsarad to work. 

or 

*First* coaplots olauss 
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(3) Firstt wash your hands. 



Page 92 
Appendix C-25 



Anaphoric Umm 

Ordinala ara aluaya anaphoric tihan thay ara intandad aa tha 
nuaarativa adjactiva aodi lying a noun but tha noun haa baan 
allipaad and tha ordinal tharafora iunctiona aa tha haad of tha 
noun phraaa* Syntax for auch a uaa would bat 



*tha* ^ ordinal (not follouad by a noun or adjactiva) 

(4a> Two conauaar-oriantad avaluation tachniquaa uara 
taatad* Tha firat tiaa triad out on auburban houaau- 
ivaa* 

which would ba raaolvad aas 

(4b> Too eonauaar-oriantad avaluation tachniquaa wara 
taatad* Tha firat conauaar-*oriantad avaluation tech- 
niqua uaa tr«ad out on auburban houaawivaa. 



Uaa Danandant on Taxt 

Ordinala uaad aa nuaarala in a noun phraaa ny or aay not ba 
anaphoric dapanding on whathar tha noun in tha phraaa haa baan 
axpraaaad any aora fully in prior taxt. In tha uaaa obaarvad in 
th^? 600 abatracta* all inatancaa of tha following ayntax whara 
thara ia an adjactiva batwaan tha ordinal and tha noun wara non- 
anaphoric uaaa. 

datarainar > ordinal <f adjactiva <f noun 

(5> Tha aacond buaiaat airport ia J.F.K. Airport in 
Naw York City. 

Thoaa inatancaa in which tha ordinal diractly pracadaa tha noun 
it aodifiaa tand to ba anaphoric but thara ara a faw axcaptiona. 
So whan tha ayntax x 

datarainar *f ordinal noun 

ia ancountaradt prior taxt will hava to ba avaluatad to aaa 
whathar tha uaa ia anaphoric or not. 

(6a> Thara had baan thraa attaapta at in vitro iarti- 
liaation. Tha third attaapt waa auccaaaful. 

which would ba an anaphoric uaa of an ordinal and would ba 
raaolvad aat 

(6b> Thara had baan thraa attaapta at in vitro farti- 
liaation. Tha third attaapt at in vitro fartilization 
waa auccaaafuK 
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Thm only trum pro-varb in thm English languag* i» thm vmrb 
*do*« In thm 600 abstrmcts analyssd* th* v«rb *do* appears in 
all 5 of ita poaaibla foraas *did«« «do'* *doaa*« ^doing* and 
*dona*« Tha rulaa for racognition of anaphoric va« non-anaphoric 
uaa of tha varb ara writtan to ancoapaaa all foraa. Vhan tha tara 
•do* ia uaad in a rula it ia to ba tntarpratad aa iaplying all of 
tha poaaibla foraa of »do'. On tha othar hand* tha nagativa con- 
tractionn of »do* will ba handlad latar in tho varbal allipaia 
of anaphora* in that tha only anaphoric uaa of thaaa con- 
tractions ia tha alliptical ona. 



Non-anapho r i c 

Tha varb *do* haa two diatinct non-anaphoric ueea: 

1. Laxical varb - aaaning »to parfora* or »to carry out*. It 
ia always tranaitiva (takaa a diract object). 

(1) Tha aubjacta did thraa aats of problaaa. 

Whan tha paat participle fora of tha verb (done) occurs in 
this usage* the sentence is in the paaaive voice and the 
diract object will precede the verb. 

(2) The aaaignaent waa done aoparately by each of the 
studente. 

When the fora »do» occura in thia uaage* the aentence ia 
frequently iaperative. 



(3) Do your hoaework! 



Periphraatic auxiliary - in thia uaage* »do» haa no indi- 
vidual aaaning but aervea aa a neceaaary verbal operator* a 
purely graaaatical eleaent which ia required for foraing 
certain caaee of a verb* or ia added aa eaphaaia in other 
inatanceae Periphraatic aeana to be foraed by the uee of 
auxiliariea inatead of by inflection of the verb. Coapare* 

(4a) She left. 



(4b> She did leave. 



*0o» aa a periphraatic auxiliary ia uaad when the aain verb 
ia in the eiaple preeent or past tense in the following 
contextas 
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• Interrogative 

(5) Did he etay long? 

• Negative 
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(6) Diatary trMtmaut did not effect total voluee 
intake. 



• Harked/ eaphatic positive 

C7) He did aek her for eoae aseietance. 

Anaphoric 

The verb 'do* hae 3 types of anaphoric loeage: 

Predicate eubetitute - the verb •do' can be ueed to replace 
a verb or verb clause. In the genre of abstracts thin use 
been observed to occur in the second of two seaantical- 
ly eontraetive elausss conjoined by a coaparative tera euch 
ae *than« or •as». and to be followed isaediately by the 
noun phraee which ie actually eubject of the verb for «hich 
*do* is eubstitutino* 

(8a) Freehaan reported leee change than did seniors, 
uhich would be reeolved ae: 

(8b) Freshaen reported leee change than eeniors reported 
change. 



Ellipsis vsrbal ellipsis is actually a special caee of 
predicate eubetitution where zero eubetitution occurs rath-* 
sr than lexical substitution. Uss of •do^ in vsrbal ellip- 
sis is dscipherable in those sentences where *do* is 
retained in its role of periphrastic auxiliary but the aain 
verb is sllided. 

(9a) I don't like cheese now but I did when I wae a 
child. 



(9b) I don't like cheess now but I did like cheeee when I 
wae a child. 

The etructural environaent differs froa that of predicate 
substitution in that 'do' is not followed by the noun 
phrase which eerves ee the subjsct of that verb clause. 

Coaplex pro-verb - when coabined with 'it', 'eo'» 'the 
eaas'f 'thie' or 'that't the resulting phrasss ('do it'» 
'do so*-, 'do ths seas' • 'do that't 'oo doing' and 'do 
this') function ae coapound referential verbal groupe which 
together replace an entire predication. 

(10) Paul woke up earlyt had a good breakfaet* and left 
on tiae for work. Ifichael did the ease. 
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INDEFINITES 



Of the 33 terns considered by Quirk I Greenbaum to be indefinite 
pronouns, 25 were observed in the set of 600 abstracts. Of these 
25 terns, 1 A functioned anaphorically at least once. Therefore, 
rules to determine whether a tern is functioning anaphorically in 
a specific instance were written for only these 14 terns. The 
table below provides sumnary statistics of the indefinite pronouns. 
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TTDMC 

1 tKno 


PsychAbs^ 


INSPEC 


TOTALS 




Ana. 


Non, 


Ana. 


Non, 


Ana. 


Non. 


F.I, 


UNI VERS ALS 
















each 


29 


28 


17 


24 


46 


52 


,47 


all 


14 


26 


7 


17 


21 


43 


,33 


every 


- 


2 


- 


4 


- 


6 


0 


everything 


- 


- 


- 


- 


- 


- 


- 


UNI VERS ALS TOTALS 


43 


56 


24 


45 


67 


101 


.40 


ASSERTIVES 
















many 


c 




I 


1 n 




VJ! 
CH 


,11 


more 


39 


49 


1 1 
* * 




50 


60 


J c 
, 40 


Bin c ^ 


2 


29 


1 


s 


3 


37 


,08 


much 


3 


4 


1 


2 


4 


6 


,40 


few 




6 




4 




10 


0 


fewer 


1 


9 






1 


9 


,10 


little 


- 


4 


- 


1 


- 


5 


0 


least 




8 


• 


6 




14 


0 


less 


10 


22 




4 


ID 


ct 


,28 


several 




9 




30 




4 n 

35 


0 


enough 




1 








1 


0 


some 


1 


26 


1 


50 


2 


76 


,02 


someone 
















sfiin^ thi no 




3 


• 


• 


• 


3 


0 




58 


180 


15 


130 


93 


310 




NON-ASSERTIVES 
















any 


2 


7 




15 


2 


22 


.06 


anyone 




1 




1 




2 


0 


anything 




1 








1 


0 


either 


20 


12 


2 


3 


22 


15 


,59 


NON-ASSERTIVES TOTALS 


22 


21 


2 


19 


24 


40 


,38 


NEGATIVES 
















no 


3 


58 


2 


13 


5 


71 


.06 


none 




1 


1 


1 


1 


2 


,33 


nothing 




1 








1 


0 


neither 


2 






1 


2 


1 


,66 


NEGATIVES TOTALS 


S 


60 


3 


15 


8 


75 


*10 


GRAND TOTALS 


128 


317 


44 


209 


172 


526 


.25 
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*Eacb* im thm Mcond indsfinit« pronoun of thm univ^rMl mub^ 
clMS to bm considttiiid. 'Each' is siailair to *all* in that its 
anajM^oric uaa can ba datarainad by ayntax only aoao of tha tiaa. 
In tba raaaining inatancaa* it ia tha prior taxt that will datar- 
a.ina uhathar ita uaa ia anaphoric or not. 

*Each' haa thraa anaphoric uaaas 

• *aach othar^ iuncviona mm an anaphoric raciprocal pronoun* 

(1) Tha aanaitixad Sa aara aora likaly to initi- 
ata convareation with aach othar than with non- 
aanai tizad Sa. 

• ^aach* functiona aa tha haad of a nosinal group and in thia 
uaa haa baan obaarvad only in tha follouing ayntactic anvi- 
ronaant: 

*aach* ^ varb fora 

(2> 117 firat-grada childran uara taatad on tha 
apparatxM and tha firat two triala coaplatad by 
aach ware racordad. 

• 'aach' ^ prepoaition othar than 'of Ca.g. at* under* within} 

(31 Sa «ara 2^ childran* 12 aach at tha tuo lav- 
ala taatad. 

Tha ona ayntactic anvironaant in which 'aach' invariably func- 
tiona non*anaphorically ia: 

'aach of 

Although tha noun phraaa following 'aach of ia itaalf frequent- 
ly anaphoric (aeg. 'each of theae' * 'each of which' >• tha tera 
'each' aervea aa a nonanaphoric quantifier aeaning 'each and 
every one of the following entities'. 

<4) Each of theae functiona ia deacribad in 
detail. 

In ite reaaining occurrencea* 'each' functiona aa a deterainer 
in either of two eyntactic environaentet 

'each' > noun 

'each' ^ adjective noun 

In theae environaenta* the prior taxt auat be conaulted to see 
whether the noun haa been acre fully apecified in an earlier 
occurrence. 58 of the 98 occurrencee of 'each' in the 600 
abatracta are of thia type which requiree acre than recognition 
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of a particular ayntax. Tha aituation ia furthar coaplicatad by 
tha fact that tba noun vhlch *aach* aarvaa aa a datarainar fort 
aay not ba tba aaaa word aa uaad originally* but rathar a paraph- 
raaa* a aaaantlcally ralatad word auch aa a aynonya or o*n*^>^l 

noun* 

(5) Surveyad 1*689 adult aarriad faaalas to axaa- 
ina aadia*-axpoaura pattarna. Each raapondant uaa 
claaaifiad aa**.* 



If tha noun that *aach' ia aarving aa a datarainar for» ia a par- 
aphraaa of* or rapatition of a aora fully apacifiad noun* than 
*aach* ia aarving an anaphoric fimction* Otharwiae* not* 

An axcaption to thia rula ia tha noun phraaa *aach S* ainca *S* 
oill ba jxKigad anaphoric/nonanaphoric in ita own right* and aach 
tharafora aarvaa aiaply aa a nonanaphoric quantifiar* 
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ALL' 



*A11* i0 mn indsfinitci pronoun o£ thm subclass tsrasd univsr- 
sal* Its basic dafinition is '*avary aaabar or individual coapo- 
nant of*. *A11* has 3 basic usass 



I. Vhan occurring in tha phrasa *all thatN tha rafaranca is 
non-anaphoric in that it is an indatarainata rafaranca to 
antitias which ara prasuaad to axist but ara not spscifisd* 
auch ths saaa as othar doubla pronouns such as *thoaa who* 
or *that which* hava indatarainata rafaranca. 



(1) All that was naadad was providsd by tha instructor. 

2. *A11* functions anaphorical ly as an indapandant noainal 
haad. in tha following ayntactic anvironaantas 



0 'all* ♦ varb fora 



(2) 13 ratardad childran and 14 childran with 
avaraga IQ*s wara tastad. All wara administarad 
tha saaa pratast* 

0 *all* prapositions othar than *of* (a.g. *but*« *un- 
iiar»f 'within' > 



(3) Tha algorithms davalopad ara all within tha 
capabilitias of tha currant systaa. 

'all' adjactiva not followad by a noun 

(4) Paradoxical lyt suggastions for aliainating tha 
dalivary ssrvicst iaproving tha sarvicat or updat- 
ing its soda wara halpful to considsr and all raa- 
sonabla froa tha financial point of viaw. 



3. 'All' aay function as an alaaant othar than haad of a noai- 
nal group* As sucht 'all' aay ba aithar anaphoric or non- 
anaphoric bassd in soaa instancas on which alaaants of tha 
noainal group follow tha tarat and in othar instancas on 
tha prior taxt« 

F i rat lyt 'all' f unc t i ona non-anapho r i ca 1 1 y when i t 
occura aa a pradatarainar/quantif iar in noainal groupa of 
tha following atructurass 



0 'all' ♦ 'of ♦ neun phrasa 



(5) All of tha tast raaulta wara distributed first 
to tha prograa coordinator. 

'all' ^ datarainar (a.g. 'tha'* 'thia'* 'such'* 
'thair') noun phraaa 

(6) All thair work was for nought. 
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0 *all* -I* adj«ctiv9 noun 

(7) All necMsary adjustBonta uara uorkad out pri- 
or to tha taat run* 

In tha following 2 noainal g^oup atrvcturaat *alP aay 
ba aithar anaphoric or nonanaphoric and tha daciaion aa to 
iihicht uill ba baaad on tha aaaantica of tha pracading 
taxts 

• •all* ♦ noun 

(8) All books uara raturnad to the library prior 
to tha nau aaaaatar. 

0 *all* nuaaral noun phraaa 

(9) All 50 atatas hava thair own ualfara aaaia- 
tanca prograaa. 

Vhara C*all' ^ noun! ia tha atructurat if thia ia tha 
firat occurranca of tha nount *all* uill ba a nonanaphoric 
quantifier and likaly a rathar generic reference* auch aa 
''all aen**. If it ia not the firat occurrence of the noun 
and the noun ia acre specified (either by preaodifiera or 
poataodif iara) in a prior occurrence in the text* then the 
uee ie anaphoric. However* if the noun ie not any acre ful- 
ly epecified in prior uae(a}* then it ia nonanaphoric. 

In that *all* ia what ia known aa a congretory quantifi- 
er* it appeara to parfora aa an anaphoric direction to 
readers to reaaaaabla and enuaarata all subgroups that aay 
hava been separated out in prior text. Thia occura aoat 
frequently when *all* precedes *Ss* or general nouna such 
as ''all groups" or "all 4 catagoriea". 

<10) 32 Ss wsra aasigned to either progreasiva relaxation 
CPR)* clinically standardised aeditation (SH)* or a wait- 
ing list control group (CG)« At the end of a 5 weak peri- 
od all Ss were expoaed to 6 vary loud tones. All 3 groups 
sxhibited higher heart rataa. 

In tha above exaaple* both "all Sa" and "all 3 groups" 
would be reaolved by reiterating the 3 groups into which 
the Ss had been subdivided* The prior text will dictate 
whether the *all* is anaphoric or not* for in aoaa inatanc- 
aa the Sa will not hava been aubdivided and therefore only 
the tera Sa is anaphoric* a. g. t 

<11) Investigated the possible influsncs of 48 hours of 
sleep deprivation (SD) in 12 19-30 year old aalaa. Fol- 
lowing SO* all Sa ahowad aarked reduction of ONA ayntha- 
sis. 



ilO 



Page 100 



Appendix C-33 

•MANY* 

•Many* im an indafinita pronoun of tha aultal aubclaaa* It 
occurrad in tha 600 abatracta a total of 27 tiaaa with only 3 of 
thaaa inatancaa anaphoric for an F«I» of .ll* 

Although thara axiat aavaral poaaaibla uaaa of *aany*f only 
rulaa for tha ona obaarvad uaa will ba includad hara ainca our 

rula-uriting ia data-*drivan rathar than thaory-drivan. 

Tha only obaarvad uaa of *aany* uaa aa an adjactiva with tha 
aaaning - **a larga but indafinita nuabar**. In thia uaat *aany* 
uaa obaarvad in thraa diffarant ayntactic anvironaanta. In tha 
firat tuof tha obaarvad uaaa uara a 1 way a nonanaphorics 

*aany* ^ adjactiva ^ noun 

(1) Daciaiona uara aada baaad on aanv pravioua caaaa. 

and 

*aany* ♦ 'of* 

(2) Halaa hava aanv of tha aaaa charactariatica aa 
faaalaa. 

In tha third obaarvad anvironaantt 

'aany' ^ noun 

tha prior taxt auat ba chackad to aaa uhathar tha noun that 
*aany' ia aodifying ia apacif iad pravioualy in any graatar 
datail. 

(3a) Raaaarch uaa conductad on a variaty of raaponaa- 
apacific atiauli. Wanv atiauli uara found to ba aora 
affactiva on iaaatura calla than on fully davalopad 
onaa* 

(3b) Raaaarch uaa conductad on a variaty of raaponaa- 
apacific atiauli. fflPY r^ffff """-^'p*" ^ * at i au 1 i 
uara found to ba aora affactiva on iaaatura calla 
than on davalopad onaa* 
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tfith analysis of tha tara 'aora'» ua ancountar for tha firat 
tiaa considaration of thoaa typaa of uorda which aarva aa cluaa 
to allipaia rathar than aarva aa anaphora thaaaalvaa* All of tha 
taraa ua ara analyzing in thia projact* whan thay function in a 
way of intaraat to ua» will function aithar aat 

!• Taraa which ara laxical anaphora* that is* placa-holdara to 
ba raplacad by taraa uaad in prior taxt. Pronouna and noai- 
nal aubatitutaa ara priaa axaaplaa* 

2. Taraa which aarva aa cluaa to tha fact that worda hava baan 
allipaad in taxt. Tha tara which aarvaa aa tha clua is not 
itsslf rsplacad* but portiona of tha prior taxt ara addad 
to tha aantanca containing tha clua word. 

'Ilora* is an indafinita pronoun of tha aultal subclaaa* which 
it aharas with 'aany** *aoat* and 'auch** 'llora' waa obaarvad in 
tha 600 abatracts a total of 109 tiaaa. 49 of thaaa occurrancaa 
wara anaphoric for an F» I. of .AS. 

In all ita uaas* praaanca of tha tara 'aora* iapliaa tha baaic 
notion that a coapariaon of soaa typa ia baing aada. Tha typa 
coapariaon baing aada will dataraina whathar tha uaa ia alwaya 
anaphoric; alwaya nonanaphoric; or dapandant on tha apacifica of 
tha taxt. 



PH TBtT 



Tha aoat coaaon coapariaon ia batwaan two clauaas. Tha co- 
oecuranca of *aora* and *than* within tha aaaa aantanca aatab- 
lishas tha nacaaaary anvironaant for elauaal coapariaon although 
*aora* and 'than* naad not ba contiguous. 'Hora** which ia con- 
aidarad tha coaparativa alaaant* togathar with 'than* foraa a 
hinga by which tha two clausaa ooalasea to fora a coaparativa 
conatruction. Tha two clauaaa ara intandad to ba aaaantic aquiv- 
alants with tha axcaption of ona alaaant which providas ths con- 
traat or coapariaon batwaan tha two clausaa* Tha two clauaaa ara 
oloaaly parallal* both in atruetura and eontant. Aa a raault* it 
is coaaon practiea to alida rathar than rapaat aoaa portion of 
what tha sacond clausa has in coaaon with tha firat clauaa. If 
thara is this allipsis* thsn for our analyais* *aora* ia to ba 
attributad with baing tha laxical triggar for tha allipaia. Tha 
tara *aora* itaalf is not rsplacad with a tara* but it aarvaa aa 
a atruetural clua that a elauaal coapariaon ia baing aada and 
that tha structurs of both clauaaa ahould ba parallal. 
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Tharsfora* whan *nor«* and *than* co-occur in a aantanoat that 
■•ntanea*a tuo clauaaa auat bm coaparad to chaok dhathar tha 
atruoturaa of tha tuo clauaaa ara ooaplataly parallal or uhathar 
yoaa taraa hava baan allipaad. If thara haa baan aoaa allipaiat 
which ayntactic itaaa hava baan alidad aay vary. For axaapla* in 
aantanoa (la) tha varb and objaot of tha aacond clauaa hava baan 
allipaadt 

(la) It uaa found that firatborna ahouad aora daath 
thraat than lataborna. 

(lb) It uaa found that firatborna ahouad aora daath 
thraat than lataborna ahouad daath thraat. 

uhila in (2a) tha aubjact and varb hava baan allipaads 

(2a) Thoaa uith dapraaaion uara aora likaly to hava 
racaivad diazapaa than antidapraaaanta. 

(2b) Thoaa uith dapraaaion uara aora likaly to hava 
racifivad diazapaa than thoaa uith dapraaaion uara 
likaly to hava racaivad antidapraaaanta. 

Nota that in raaolving tha allipaia tha tara *aora* ia not car- 
rlad foruard and ra-uaad uith tha othar taraa in tha 2nd clauaa. 

Kouavar* it ia not to ba aaauaad that all aantancaa uith co- 
occurrancaa of 'aora* and *than* hava aoaa alaaanta alidadt but 
rathar tha praaanca of thoaa taraa raquiraa that tha atructura ba 
chackad for axact parallal conatruetion. 

It doaa occur aoaauhat infraquantly (4 out of 46 allipaaa) 
that tha allipaia appaara to ba both cataphoric and anaphoric* 
uith aoaa uorda froa prior taxt and aoaa uorda froa latar taxt 
u a ad to flaah out a eoaplataly parallal atructura. Va uiil con- 
aidar thaaa occurancaa anaphoric in that both tha anaphoric and 
cataphoric allipaaa naad ba raaolvad. 

(3a) Tha axaaplaa givan indicata that youngar Sa aada 
aora falaa than trua concluaiona. 

(3b> Tha axaaplaa givan indicata that youngar Sa aada 
aora falaa concluaiona than youngar Sa aada trua con- 
cluaiona. 



Vhan coaparing thm clauaaa for parallal atructura* all othar 
anaphora auat ba raaolvad firat to inaura that tuo diffarant 
uorda ara not craditad uith eraating tha aaaa alliptical aitu- 
ation. Thia ia particularly iaportant if tha varb of tha aacond 
coaparativa clauaa ia a fora of tha provarb *do*« aa aaan in t4a) 
uhara tha varb *did* functiona aa a pradicata aubatituta for tha 
antira varbal clauaa. In thia aantanca* tharafora* *aora* uii: 
not ba conaidarad a clua to anaphoric allipaia. 
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(4a) Fourth-gradiarB mmdm significant ly aora faaala 
daaignationa aaong adult-apaeif iad faaalaa than did 
praaehoolara. 

(4b> Fourth-gradara aada aignif ieantly aora faaala 
daaignationa aaong adult-apacif iad faaalaa than praa- 
ehoolara aada faaala daaignationa aaong adult- 
apaoifiad faaalaa. 



tfhan uaad aa a quant i far* *aora* aaana '*an additional aaount 
of thinga* paraona* tiaa. ate." and diraetly praeadaa tha noun 
phraaa it aodifiaa. 

*aora' -*- noun phraaa 

(5) Raaulta ahou aora aaphaais on tha inforaational 
aapacta. 

tfhathar tha uaa ia anaphorie or not uill dapand on whathar tha 
noun phraaa it ia aodifying ia apaeifiad any aora axtanaivaly in 
prior taxt. 



AMAPHORIC 



tfhan a eoapariaon ia aada batuaan an abaoluta nuaarie valua 
and itK coaparativa fora <a. g. 'two or aora than tuo*). tha taxt 
ia fraquantly abbraviatad to? 

nuaaral 'or aora* ■^ noun/adjactiva noun 

Thia uca ia a elua to anothar inatanea of anaphorie allipaia in 
that *than -•' nuaarcl' hava baan allipaad. 

(6a> Tan or aora inataneaa of tardinaaa uill raault 
in auapanaion. 

vhieh would ba raaolvad aas 

(6b> Tan or aora than tan inataneaa of tardinaaa uill 
raault in auapanaion. 
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Vh«n thm comparison is bsing asds bstussn soas sntity and an 
sxpliclt standard rathsr than bstuaan two clauaast 'aora* will 
diractly pracada *than* and ba follouad by aoaa spacific nuaaric 
aaaaura. 'Nora* ia navar anaphoric in thia uma. 

C7> Tha avaraga baar uaigha aora than ."2000 pounda. 



InttTwIlttr 

*nora* is usad as an intansifisr to fora tha coaparativa fora 
of both adjactivas and advarbs which it praaodifiaa* Tha adjac- 
tiva or advarb aust ba of tha o>**dabla typa* that is* it aust ba 
an attributa that aay ba prasant to varying dagraas* Vhan func- 
tioning as an intansifisr* *aora' is nonanaphoric. Tha ayntactic 
anvircnaant in which this uss of *aora* is found is sithsrs 

*aora* ^ gradabla adjactiva 

(8> Patiants with low IIHPG lavals ara aora raaponsiva 
to traataant with drugs that inhibit norapinaphrina 
uptaka« 

or 

*aora* ^ gradabls advarb 

(9> Changa towards incraasad assartivanaaa is aora 
liksly to occur whan cliants raalistically aaaass tha 
possibilitiss opsn to thsa. 

Tha intanaifiar tias occurs only in thoaa ssntancas in which 
*than* doss not co-occur with *aora*« Evan if *aora* praaodifias 
a gradabls adjactiva* if *than* is also prssant* tha uss of 
*sors* is to bs catagorissd as a clauaal coaparativa. 

Tha rssson *Bors* without *than* cannot bs intarpratad as a 
laxieal elua to allipais* is that sines ths writsr did not indi- 
csts by uss of *thsn* on what paraaatar tha coaparison waa to 
taka placa* thara is aora than ons intarpratation possibla and us 
oannot a as u ss what was intsndad. For instanca in tha following 
pisca of taxt* ths coaparison is aabiguous bacauss thara is no 
*than*. 

(10> laagary thtvory is aors of s thaory of problaa 
solving and is bast aicaainad through tha aaaaura of 
srror rata. Linguistic thaory is laxft • aaasurs of 
santanca procassing and is bsst saaaurad using latan- 
ciss. 
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•BOST» 

'Host* is an indsfinits pronoun of thm multal subclaBS. 'no«t' 
ocourrttd 40 tinas in thm mmt of 600 abstrscts ulth only 3 of 
thaaa occurrancaa baing anaphoric for an F.I. of .075. 

In that floaa uaaa of *aoat* ware so infraquont as to be singu- 
lar in thsir occur rancQt rulss havs not been developed for all 
observed ussst but rather* in soae instances the observed syntac- 
tic environssnt is simply describsd. 



*Nost* has four basic functions^ 



Supsrlstivss 'Host' is used to create the superlative form of 
both adjectives and advsrbs. Ths seaning of *aost* in such 
instsncss is **to ths grsatsst or highest degree**. When ueed to 
fora the supsrlative of an adjc^ctive* one basic syntactic envi- 
ronaent would be: 

»the» ♦ *80St» ♦ adjective ♦ noun 

(1) Short-tera instabilities are the aost important 
sourcs of srror. 

in such m syntactical contexts 'aost*' is non-anaphoric. Ifw hoci- 
svsr* ths sdjective is not followed by a noun: 

•the* ♦ 'sost' ♦ adjective 

the term *BOst' is to bs considsrsd anaphoric in that it serves 
ss s Isxicsl clus to ths sllipsis of ths noun. 

(2) Six snvironasnts wsrs testsd for conducivenees to 
study* Low hsat and high light were found to be the 
most conducive. 

Another syntsctic snvironssnt for 'most* when it forms the super- 
Istivs of sn sdjsotivs is basicslly ths same as that used by 
*sost* to form ths supsrlativs of an advsrb* and in all instances 
it is non-snaphoric. 

vsrb •to be* ♦ 'aost* ♦ adjective/ adverb 

C3I To detsrsins which of ssvsral methods was most 
ftffM:tivs» s ssriss of tssts was run. 

(4) Short-answsr questions are aost often inappropri- 
ately answered. 



Qusntifisr: 'Host* is ussd as an indsfinits quantifier of mass 
nouns snd plural count nouns* whsrs its seaning is* respectivs- 
ly* ''orsatest aaount of** and **greatest number of**. In such 
usesf *sost' is distinguishsd fros its superlative ijee* by the 
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fact it im not pr«c«d«d by *th«***Kost* is nonanaphoric uhan it 
occurs in aithar of tha two aoat fraquant syntactic environ- 
santo for ^aost* as a quantifier: 

*aast* noun/ ad j active noun 

(7) By following tha Pritkin diatt aoat overueight 
teenagers lost 10-15 pounds* 

or 

•sost^ + 'of^ 

C8> Host of tha etudents passed the final exam. 

However* if tha noun which *aost* is quantifying is allipeed* the 
use is anaphoric* 

*aost* -I* vsrb 

(9) Fifty attendees were bunked together. Host 
.enjoyed the experience. 

Mount *no8t* was obssrved once in its use as a noun* where the 
asaning is **ths greateet aaount*** as distinguished from ite 
ssaning aa a quantifier - ''the greatest aaount of Its syntax 
let 

•ths^ + «aost 

(10) Its ths sost I can gst for the car. 

Adverbs *Ifost* say itself be ueed as an adverb* not just to fora 
the supsrlstive of an adverb. In euch uee* *aost* is non- 
anaphori-::^ and has bssn obssrvsd ones in each of the following 
syntactic snvironaents: 

• As sn interposing eleaent cauaing a split infinitive: 

•to' ♦ 'sost* ♦ verb 

(5) The drug was shown to aost effect results in pre- 
aature babies* 



• As s displaced adverb: 

verb direct object 'nost' 
(6) She baked pies aost during the winter months. 
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•MUCH* 

*nuch* im another of thm indttfinita pronouns of thm aultal 
vubclaM. *nuch* occur r«d m total of 10 tiaas in tha 600 
abatracta* with 4 of thaaa occur rancaa baing anaphoric for an 
F.I* of .AO. *IIuch* waa obaarvad in 4 diatinct uaagaa. Tha firat 
uaa ia dapandant on prior taxt to dataraina whathar it ia ana- 
phoric or AOt» tthila tha lattar 3 uaaa ara nonanaphoric in all 
obaarvad inatancaa* 

• Clauaal coaparativa 

Vith analyaia of *auch* ua ancountar a 2nd tara which fra- 
quantly aarvaa aa a laxioal olua to tha fact that aoaa worda. 
in taxt hava baan allipaad. 'Mora* ia tha othar tara which 
parforaad tha aaaa function. Both taraa ara uaad in coaparins 
two olauaal conatruettona which aro aaaantically parallal. 
Sinoa tha 2nd clauaa* if fully flaahad out* would ba a ayn-* 
tactic duplicata of tha firat clauaa^ it ia coaaon practica 
to allida rathar than rapaat aoaa portion of tha coaaon 
atructura. 

Tha ayntaotic anvironaant in which *auch* functiona aa 
thia laxical clua to allipaia imt 

*auch* ^ adjectiva *than* 

or 

*auch* ♦ advarb ♦ 'than* 

*Than* aay or aay not iaaadiataly follow tha adjactiva or 
advarb* but tha praaanca of *than* ia aaaantial to xndicata 
that in fact a eoapariaon ia baing aada. 

Vhan thaaa particular ayntactical anvironaanta ara ancoun*- 
tarad in taxt* it ia nacaaaary to chack whathar tha at rue- 
turaa of tha two clauaaa ara eoaplatalv parallal or whathar 
aoaa taraa hava baan allipaad. 

(ia> Firat-borna raapondad to tha anxiaty atiaulua 
lUSlL dif farantly than latar-borna. 

(lb) Firat-borna raapondad to tha anxiaty atiaulua 
auch diffarantly than latar-borna raapondad to tha 
anxiaty atiaulun. 



In thoaa inatancaa whara aoaa taxt haa baan allipaad in tha 
aacond coaparativa clauaa* tha uaa of 'auch' will ba conaid- 
arad anaphoric* whila if no taraa hava baan allidad* tha uaa 
ia non-anaphoric. 

Tha ona axoaption ia whan 'auch' praaodifiaa anothar tara 
of tha olaaa of indafinita pronouna which ia itaalf baing 
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UMd to form thm eonpanitive fora of thm «djTCtiv« or advorbt 
g. 

*nueh* ♦ 'ftors* ♦ adjective ♦ •than^ 

(2) Inaptltuda um aueh motm diffloult to pratMt for 
than uaa diaintaraat. 

Vhan doubla indafinitaa occur* tha firat indafinita ia to ba 
thought of aa an intanaifiar and non-^anaphoric in all occur- 
rancaa* uhila tha aaeond indafinita pronoun will ba attribute 
ad with baing tha laxical triggar for allipaia. 

Intanaifiar 

*nuch* oparataa aa an intanaifiar whan it pracadaa an 
adjactiva but tha elauaal hinga *than' ia abaant froa tha 
construction. In aueh a uaa* *auoh* ia non-anaphoric. Tha 
syntactic anvironaant would bas 

*auch* adjactiva noun 

(3) Earliar in hia caraart Vataon had auch loftiar 
goala. 



Ad J activa 

*Ruch* can alao function aa a aiapla adjactiva with tha 
aaaning ''graat in quantity* aaount* axtant* or dagraa**. Such 
a uaa ia nonanaphoric and waa obaarvad onca in tha following 
ayntaxs 

*auch* ♦ noun 

(4) Thara ia auch truth in what you aay. 

Noun 

*nuch* waa alao obarvad onca in ita nonanaphoric rola aa a 
noun in tha following contaxts 

varb *f *auch* * infinitiva clauaa 

(5) Hia axcuaa laft auch to ba daairad. 
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Thm o^oup of ind«finit« pronouns knoun mm thm paucal vubclaes* 
consists of thii tarsBi faut fauar» fauastt littlat laaat and 
laaat. 

This sx'oup was diatributad in tha 600 abatracta as folloua: 



Paych Aba Inapac Total 

Ana* Non. Ana* Non. Ana. Non. F« I • 

fau - 6 - 4 - 10 0 

fa^^ar 1 9 - - 1 9 .10 

fauaat - ^ • • - - - 

littla - 4 - I - S 0 

laaa 10 22 - 4 10 26 .28 

laaat - 8 - 6 - 14 0 



A full linguiatic analyaia » including rula-taatingt uill ba 
parforaad only on tha tara Maaa*» and tha aingia anaphoric 
instanca of 'fauar' aiaply daacribad. 



'Fauar* 

Tha singla anaphoric uaa of ' f auar' occurrad in a aantanca 
coapoaad of tuo aaaantically parallal clauaaa* tihara tha co- 
occurranca of 'fauar* and *than' providad tha ayntactic anviron- 
aant paraitting aoaa laxical alaaanta of tha aacond clauaa to ba 
allidad. Tharafora* 'fauar' aarvad aa a laxical clua to allipais 
and ia attributad with anaphoric status. This is tha aaaa baaic 
uaaga obaarvad with tha othar tuo paucal coaparativass *aora* and 
Maas*. 



ERiC 



120 



Page 110 
Appendix C-43 



*Lm0* is mn indsfinite pronoun of the paucal subclass. It 
occurrsd 36 times in the 600 abstiacta with 10 of these uses 
being anaphoric for an F.I. of .28. 

'Less' has 4 specific uses. Tuo of these uses may be anaphoric 
or not depending on prior textt while the other tuo uses are 
aluays nonanaphoric. 



PEPEMPEMT QM TEXT 



Vhen Mess' and 'than* co-occur in a laentencet the environment 
exists for a comparison to be made betueen tuo entities or tuo 
clauses. If the comparison is betueen clausest the common prac- 
tice is to ellide some portion of the second clause uhich is sim- 
ply a repetition of elements of the first clause. If this type 
ellipsis occurst Mess' is to be attributed uith being the lexi- 
cal clue for ellipsis and therefore anaphoric. If« houevert the 
second clause is completely parallel uith the first and no words 
have bean ellided* the use of Mess' is nonanaphoric. 

The syntax for such a comparative use is: 

'less' + + 'than' 

uhsre what occurs betueen the 'less' and the 'than' is highly 
variablSf but ths presence of both predicts this usage. Vhen 
this syntax is encounteredt the second clause must be checked for 
cosplete syntactic parallelism uith the first clause. 

(la) Firstborns reported less death-threat concern 
than other groups. 

uhich would be resolved as: 

(lb) Firstborns rsportsd less death-threat concern 
than other groups reported death-threat concern. 



The one exception to this ruls is the idiomatic phrase 'less 
than' foUoued by some adjectival forst e.g.: 

(2) He uas less than honest. 

where the true seaning of 'less than' is **by no means**. The syn- 
tactic environment for this exceptional use is: 



121 



Page 111 
Appendix C-44 

*l«as than* ^ adjective 

If Mmb than* io not follouad by an adjactivet it is to ba 
traatad tha aame aa in tha clauaal coaparativa usage and tha 
clauaa that follows Maas than* ia to ba examined for complete 
parallel structure with the first clauaa. 



Quant it ier 

Vhen ueed as a quantifiart *lees* precedes a noun phraset 
which aay conaist of either: 

* lees* + noun 

or 

*less* ^ adjective noun 

and the term *than* does not co-occur. In this usaget however* 
the adjective will not be of the gradable typet which it ia in 
the negative coaparativa use. Vhether the uee is anaphoric or not 
will again depend on whether the noun that *less* modifies is 
spscifisd any more (extensively in prior text. 

(3) Experienced programmere required lees uarm-up 
tiae to score highly. 



MOII-AMAPHORIC 



Meoative cosparative 

*Lese* cosbines with gradable adjectives and adverbs to form 
their negative comparative form. Gradable refers to an attribute 
that aay be pressnt in varying degrees. In this usage* * than* 
never occurs in the conetruction* which consists of: 

*less* ^ adjsctive 

or 

* lees* ^ adverb 

(4) Urban lota are considered to be less stable in 
the current real estate market. 
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ADVERB 

Aa an advarbt Maaa* aervea aa a **dountoner**t lowering tha 
affaet of tha forca of the verb. Tha ayntactic environment for 
auch uaaga would be: 

verb * leaa* 
or 

verb ^ direct object ^ *leaa* 

In aoaa of ita other uaeat * laaa* aay also follow tha varbt but 
in thoaa uaaa* * laaa* would be followed by either an adjectivet 
advarbt nount or 'than*. Vhen uaed aa an advarbt Maaa* ia not 
followed by any of theae* and either anda tha aentence or ia fol- 
lowed by a prapoaitional phraae. 

(5) Students cheated leaa when dual monitoring 
davicea were uaed. 



•SOME* - group 
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Of tho 'soM' group of ind«finit« pronouns* 'soasbody' and 
'ttoaaona' navar occur in tha aaapla aat of 600 abatracta* uhila 
'aoaathing' only occura 3 tiaaa and ia non^anaphoric in aach 
uaaga. Only *aoaa* waa obaarvad in any anaphoric uaaa. 'Soaa' 
occurrad 78 tiaaa in tha 600 abatracta* with only 2 of thaaa 
inatancaa baing anaphoric for an F.I» of •02. 

Mon--anapho r i c 

Tha aajor rola of *acaa* ia to aarva aa a quant ifiar/ 
datarainar of a noun phraaa. In auch uaaga* ita aaaning ia **an 
unapacifiad aaount or nuabar**. 'Soaa' aay iaaadiataly pracada tha 
noun phraaas 

'aoaa' noun phraaa 

(1) Soaa coaputar^aidad daaign prograaa ara daacribad 
and illuatratad uith axaaplaa* 

or taka tha of-conatructiont 

'aoaa' *of' noun phraaa 

(2) , Each araa la daacribad datailing aoaa of tha 
aajor propoaad aolutiona to tha propoaad tharain. 



Anaphoric 



It ia poaaibla for tha noun phraaa which 'aoaa* ia aarving aa 
datarainar for* to ba allipaad. In auch a uaaga* 'aoaa' ia ana* 
phoric. Tha poaaibla ayntactic anvironaant for auch a uaa uould 
ba aithar: 

'aoaa' varb phraaa 

or 

*aoaa' prapoait^.on (othar than »of') 

(3) Tha anauara wara incorract for a nuabar of raa* 
aona. Soaa wara incoaplata and foaa aiaply wrong. 
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group 



Of thm *«ny* o>^oup of ind«finlt« pronouns* 'anybody* n«v«r 
oceurrad in thm saapla Mt of 600 abstracts. *Anyona* only occurs 
tulcs and * anything* oncs« Nona of thssa occurrancas ara ana- 
phoric. *Any* occurrad 24 tiaaa* uith only 2 on thaaa instancas 
baing anaphoric for an F.I* of .OS. 



*Any* sarvaa as a quant if iar/datarainar of a noun phraaa and 
ths quaation of uhathar tha usaga is anaphoric or not ia anauarad 
only by axaaining prior taxt to saa if tha noun that *any* is 
aodifying is spacifiad aarliar in any grsatar datail. 
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•EITHER^ 

*Eith«r* occurred in thm 600 abetracte a total of 37 tlaas 
with 35 of thaaa occurrancaa baing anaphoric for an F.I. of .95. 
4 of tha 35 anaphoric occurrancaa uara laxical anaphora and tha 
^•Mining 31 anaphoric uaaa of *axthar' uara as laxical cluaa to 
allipaia. 

mmm<; 

Coordination 

Tha aajor function of tha tara «aithar« ia aa an anticipator 
of a coordinatad eonatruction in which tha actual coordinator 
tara ia 'or*. 'Eithar - or* aay ba uaad to coordinate within 
phraaaa or acroaa phraaaa and clauaaa* and in both anvironaanta 
'aithar* ia conaidarad a laxical clua to anaphoric allipais. 

W>»«**1 Coordinations Tha uaual ayntax for within-phraaa coordi- 
nation ia aithar: 

*aithar« ♦ adjactlva •or* ♦ adjactiva ♦ noun 



(la) Subjacta dalivarad a praparad apaach on githar a 
aaxual sx. * non-aaxual topic. 



which would ba raaolvad aa: 

(lb) Sub j acta dalivarad a praparad apaach on aithar a 
aajcual topic or aubjacta dalivarad a praparad apaach 
on a non-aaxual topic. 

and parhapa aora naturally raphraaad ao: 



(le) Eithar aubjacta dalivarad a prepared apeech on a 
aaxual topic or aubjacta dalivarad a praparad apeech 
on a nonaaxual topic. 

or 

fora of verb •to ba^ ♦ •either^ ♦ adjective ^ 'or' ^ adjective 

where the attribute* expreeeed by both adjectivee are being 
predicated of the aaae noun phraae which precedes the verb fore 
of 'to be*. 

(Sa) Stiauli were either aweet or aour. 

which would be resolved aas 

(2b) Stiauli were either aweet or atiauli ware eour. 

and aora naturally raphraaad aas 
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(2c) Either stiauli umtm «UMt or stiaull umtm sour. 

ClauMl eoordinmtions All oth»r co-occurrancM of *sither* and 
*or* which do not fit ths two syntactic snvironaents dascrib«d 
abovs* sill ba instancss of clausal coordination. Typical usa 
sight bss 

(3a> Tha diaaaaa aithar raspondad paradoxically to traat- 
ssnt or continuad to prodiica savara ayaptoas. 

which would ba raaolvad as: 



(3b> Tha diaaasa aithar raspondad paradoxically to 
traatssnt or ths disaasa continuad to produce ayap- 
toas. 

snd Bors naturally raphraaad aas 

(3c) Eithar tha disaass raapondad paradoxically to traat- 
ssnt or ths dissaaa continusd to produce sysptoas. 



Dstsrainar 



'Either* aay ftmction as dstsrsinsr of s noun phrase and is 
always anaphoric in such usage. The environaent for such use 
would be the non-occurrence of the ters *or* within ths sase sen- 
tence snd ths syntsx: 



*sithsr* 4^ noun phrase 

icond 
_ithsr 

icant aids sffscts. 



C4) In ths sscond sxpsriasnt. codeine and daaarol 
<Mre teated. Either drug was found to produce aignif- 
int side effects. 



MOM-AlfAPHORIC 

'Either* say function as s noainal* ssaning **ona or the oth- 
In such s use* *sithsr* has been nonanaphoric in each 
occurrence in the test sst» «hsn functioning aa a noainal* 'ei- 
ther* occurs in a ssntsncs without *cr* and in tha followina ayn- 
tax» 

'either* > 'of ♦ noun phraae 
(5} Subjects were placed in either of two conditiona. 




127 



Page 117 
Appendix C-5Q 



TlMrtt mrm tivm n«gstiv« indafinits pronouns - *no*. *non«*t 
'nobody'* 'nothino' and 'naitter'. In thm mmt of 600 abstracts 
tbsM tarM oceurrad aa folloass 



'nobody' - no occur rancaa 

'nothing* • ona non*anaptaoric occurranca 

'nona' - 3 occur rancaas 1 anaphoric • 2 nonanaphcric 

'nsithar' - 3 occur rancaai 2 anaphoric* 1 nonanaphcric 

'no' • 76 occurrancaai S anaphoric* 71 nonanaphcric 



Rulaa for 'nona' and 'naitbar' can ba aaaily ganaratad froa 
aarliar rula aata vrittan for aiailarly functioning taraa. 'Mo' 
ia tbs only nagativa vhich oceurrad sufficiantly fraquantly to 
aarrant a full**acala analyaia* 



'MGHE' 

'Vona' had too diatinct usaa in tha abatracts. Tha rulaa gov- 
arning vhathar tha uaa oaa anaphoric or not ara tha aaaa ayntas- 
aatching rulaa aa uaad for tha taraa 'aoat'* 'all' and 'aach'« 

'>ona' ia non-^anaphoric in tha ayntaxt 




'nona' ^ 'of ^ noun phraaa 



CD Vona of tha aaaay ({uaationa oara raapondad to in 
aufficiant datail. 

'Mona' aarvaa aa a clua to anaphoric allipaia in tha ayntax: 

'nona' ♦ varb fora 

C2> Thraa indaxing tachniquaa vara taatad. Nona 
iaprovad tha raaulta significantly. 



'MEiTHEH' 



Tha tso diatinct uaaa of 'naithar' vara axact ayntactic aatch- 
as to too of tha uaaa that 'aithar' ia put to. Waaaly* 'naithar' 
ia uaad aa a datarainar and ia anaphoric in tha ayntaxs 

'naithar' '•^ noun phr 



(3> Subjacts osra aasignsd to a control group or tha 
axpariaantal group* Kaithar group parforaad axcap- 
tionally oall* 
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Page 118 
Appendix C*51 

'Nsithar* im xmmd mm m noainal with th« ManinEi "not on* or th« 
otlMr" and is nonanaphoric in tha ayntaxs 

'naitliar* *of* noun phrasa 

(4> Waithar of tha firaa raaultad in any loaa of 
lifa. 
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Page 119 
Appendix C*52 



•M0» 




*Ko* occurred a total of 76 tiaaa in tha 600 abatracta with 
only 5 of thaaa occurrancaa baing anaphoric for an F.I. of .oe. 

Tha ona poaaibly anaphoric umm of *no' ia dapandant on prior 
taxt« Tha ayntax for auch uaa would bmz 



ohara tha anaphoric/nonanaphoric daciaion dapanda on uhathar tha 
noun that 'no* ia aarving aa datarainar for ia apecifiad in any 
Oraatar datail aarliar in taxt. 

€la) Thraahold-raiaing tachniquaa hava baan undar 
davalopaant for aavaral yaara. Mo tachniquaa hava yat 
aat tha daaign critaria* 

ohich aould ba raaolvad aas 

fib) Thraahold^raiaing tachniquaa hava baan under 
davalopaant for aavaral yaara. No thraahold-raiaing 
tachniquaa hava yat aat tha daaign critaria. 



'Mo' ia als»ya non-anaphoric uhan praaodifying either an 
adjectives 



€21 Mo aignificant affecta ware found for birth- 
atatua alone* 

or an adverb < 



(3) Phyeiciana believe that quarantine ia no longer necea- 
aery for victiaa of tuberculcaxa. 



•no* ♦ noun 




'no* ^ adjective ^ noun 



•no' ♦ adverb 
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Appendix C*53 

Rssldual Adj«ctiv«s 




TIw 6 rsuining P«A**« that function froqiNintly mm adJsctivM 
umrm analyssd uith thm fol lowing rMUltst 



P* A. I MSPEC TOTALS 



t«ra 


Ana. 


Hon. 


Ana. 


Hon. 


Ana. 


Non. 


F. I. 


additional 


0 


2 


1 


3 


1 


5 


.16 


anothar 


2 


3 


1 


3 


3 


6 


.33 


both 


18 


36 


1 


25 


19 


61 


.24 


alaa 


0 


0 


0 


1 


0 


1 


.00 


aqual 


0 


5 


0 


1 


0 


6 


.00 


idantical 


0 


2 


0 


1 


0 


3 


.00 



Sinca 'alaa'. •aqual'* and 'idantical* navar functionad ana- 
pborieally and 'additional' <unctionad anaphorical ly only onca. 
no furttaar daacription of thair uaaga »ill ba praaantad. nor Bill 
thay ba tastad. 



'Another' aay tunction in ona of thraa uayas 

Vofk-anapihories 'Anothar' ia aloaya non-anaphoric nhan xjmmd. to 
rafar to aoaa indatarainata huaan rafarant who ia praauaad to 
•xiat but not apaeifiad in tha taxt. 



CDForgivonaaa of anothar bringa panca of aind. 



Ikapandaiit-on-tax t f 'Anothar' ia potentially anaphoric whan it 
■arvaa aa aodifiar in a noun phraaa. Vhathar it ia anaphoric or 
not dapanda on whathar tha noun it aodifiaa haa baan apaeifiad 
in eraatar dotal I aarllar in taxt. 

C2} Thara ara a variety of ballet etylaa currently in 
voflua. One ballet etyle ie the elaaaical and anothar 
atyla ia tha ainiaaliat. 



Aaaphoriei 'Another' ia aluaya anaphoric when the noun it ia 
intended aa aodifiar for. haa been ellipaed. 

(3} It haa becoee inereaainoly difficult to tell one 
book froe another. 



Both 

'■otb' baa 2 non-anaphoric uaee and 2 anaphoric uaee. 
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Appendix C-54 



Thm «O0t ooMon xmm of 'both^ is in conjunction with •and* in 
uhmt im known mm m cowbinatory coordination* •Both* ia uaad to 
atraaa tha incluaion of aach of tha 2 worda or phraaaa baing 
eoordinatad. Tha occurranca of tha following ayntax alwr.ya indi- 
cataa thia uaat 

•both* 'and* 

whara tha taxt which aaparataa tha 2 taraa aay ba aa abort mm ona 
word or aa long aa a full phraaa. 

(4) Both tha autoaaton and ita ravaraal ara atrongly 
eonnaetad* 

or 

C5) Conatructiva aaaartiva al tarnati vaa ara davalopad 
that Intagrata both tha taak and faalinga. 



Vhan *both* coabinaa with *of*» it again mtrmmmmm incluaion of 
aach of tha itaaa which follow *of*. *Both* im alwaya non- 
anaphoric in auch uaa» although tha tara or phraaa following *of* 
ia fraqpiantly anaphoric. 




•both* ♦ 'of* 



€6) Both of thaaa tachniquaa hava baan uaad in aarli- 
ar raaaarch in contant analyaia* 

Anaphorie 

• Bo th^ waa obaarvad to f unc t i on anapho r ica 1 1 y in mvmry 
inatanca wbara it aarvad as praaodifiar in a noun phraaa* TMa 
uaa can ba racognixad by abaanca of •and^ froa tha conatructicn 
a^ ona of tha following syntactic pattarnax 

•both^ ♦ noun 

(7) Rata and garbila wara taatad in tha aaaaa. Both 
apauiaa iaprovad parforaanca following rainforcaaant 
triala. 

•both^ ♦ adjactiva ^ noun 

(8> Pra-adolaacant faaalaa and adolaacant aalaa wara 
obaar^rad in thair school sattings* Both targat groupa 
axhibitsd sal f -conscious bahavior whan adviaad of tha 
poaaibls obsarvationa. 




*Both* funotiona anaphorically whan it sarvas aa a pronoainal* 
taking tha placa of two itaas rafarrad to aarliar in taxt. In 
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Appendix C-55 



this usaget 'both* occurs uhsrsvsr a noun sight occur and has 
bssn cbssrvsd in ths following tuo pattsrnst 




•both* ♦ vsrb 



(9) Rsd and ysllou usrs chossn as ths stisulus col- 
ors. Both slicit sisilar ssotionsl rssponses in sub- 
jscts« 

prsposition ♦ •both* (not follousd by adjsctivs or 

noun) 

(10> Hsavy saoksrs and frsqusnt drinksrs usrs chosen 
ss subjscts. Lsck of intsrsst in nutritional concsrns 
has bssn obssrvsd in both. 
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Appendix C«56 

AdvarbB 



Thm linguist Ic analysis of ths 10 advsrbs which occur in ths 
sst of 600 abstracts has bssn coaplstsd uith ths following suaaa* 
ry rssultss 



P. A. 
Ana. Non. 





0 


2 


idantlcal ly 


0 


0 


•iBil«rly 


1 


1 


■o 


3 


8 


■uch 


16 


17 


than 


1 


14 


th«r« 


0 


36 


th«r«in 


0 


0 


thtjs 


0 


10 


vies versa 


1 


0 


Total 


22 


88 



Ina. 




Total 


Ana 


Wan 


Ana. 


Man. 




1 


2 


1 


4 


.20 


0 


1 


0 


1 


0 


0 


0 


1 


1 


.50 


0 


15 


3 


23 


. 12 


20 


27 


36 


44 


.45 


1 


24 


2 


38 


.05 


1 


24 


1 


60 


.02 


1 


0 


1 


0 


1.00 


0 


3 


0 


13 


0 


0 


0 


1 


0 


1.00 


24 


96 


46 


184 


.20 



As can bs sesn by thsss figurss* only *so' and 'such* dssand that 
rulss bs arittsn to dstsrsins anaphoric fros nonanaphoric occur- 
rsncssa *Thsn*» with 2 anaphoric occurrsncss could possibly havs 
rulss srittsnt but ths singls occurrsncs in sach databass doss 
not sppsar to of far any pattsrnsd uss* 




134 



Page 125 
Appendix C«57 

•SO* 

*So* occurred a total of 26 tiaaa in tha 600 abatracta. 15 of 
thaaa occurrancaa uara in tha INSPEC abatracta and all 15 occu- 
rancaa uara non-anaphorio* Of tha 11 occurrancaa of *ao* in PSYCH 
ABS» 3 inatancaa uara anaphoric. Total F* I* ovar 2 databaaaa uaa 
12%. Rulaa will ba urittan baaad on tha uaaa of *ao* in Juat tha 
PSYCH ABS databaaa. 

Non-Anaphoric 

!• *So* combinaa uith *that* to introduca a clauaa axpraaaing 
purpoaa or raault* Syntax would bas 

»ao» ♦ »that» 

(1) Tha valocity of a trollay auat ba control lad ao 
that tha auinQ of ita o^^b vaniahaa whan tha trollay 
arrrivaa at a goal poaition* 

2. *So* coabinaa with foaa of tha varb 'do' to fora a coaplax 
anaphoric pro-varb. For thia tabulation of anaphoric taraa* 
'ao' in auch uaa «iill not ba eountad mm anaphoric ainca in 
aach inatanca« 'do' haa alraady baan craditad uith anaphor- 
ic function* Raaolution of tha coaplax pro«*varb 'do' ra- 
inaarta thoaa taraa uhich 'ao' aubatitutaa for* 

fc2»a of 'do' ♦ 'ao' 

(2) Paul haa alrafttdy k'agiatarad for tha nau aaaaatar 
and Gana uilt do ao aoon* 

3* 'So' functiona non-anaphorical ly aa an intanaifiar of 
aithar an adjactiva or advarb* and haa tha aaanino ''to a 
graat axtant or dagraa"* Racognisabla ayntax would ba: 

'ao' adjactiva/sdvarb 

(3) Tha childran umrm ao aagar to bagin that to uait 
uould hava baan fooliah* 



Anaphoric 

In all raaaining obaarvad inatancaa of uaa* 'ao' functionad aa 
a prn**advarbial* In mtich uaaa» ita aaaning ia **auch aa haa baan 
apacifiad or auggaatad** aarliar in taxt* Tha contaxtual ayntax of 
auch tjaa waa variad in that 'ao' can raplaca an advarb or a uhola 
clauaa* 

(4) Thay aakad uhathar ua wara going to tha concart* 
If ao» thay uantad to go with ua* 
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rage ico 
Appendix C«-58 



SUCH' 



*Such* occurred in thm mmt of 600 abstracts a total of 80 
tiass uith 36 of thssa occurrsncss bsing anaphoric for an F. I. 
of 45%. *Such* has 2 consistsntly non-anaphoric usest 2 consis- 
tsntly itnaphoric usss* and 1 uss depsndant on text* 

NON-ANAPHORIC 

1. *Such* coabinss with *that* to foxa a compound subordinat- 
ing conjunction introducing e clause. It is aluaya non- 
anaphoric in the syntax: 

*such* ♦ •that* 

(1) The results uere pressntsd in s aanner such that 
those unfasiliar with the topic still had no diffi- 
culty understanding thea. 

2. *Such* coabines with *as* to serve as an explicit indicator 
that an appositional phrase follous* The appositional 
phrase provides one or sore exaaples of the noun phrase 
that precedes it. 

(2) Skills such as providing syapathy* explanation 
and advice are given. 

in which case the eyntax would bes 

noun ^ *such' •as* 
A possible alternative syntax would be: 

*such* ♦ noun ♦ 'as* 

(3) The baslcetball teaas in contention for first 
place are such teaas as Georgetown* Syracuse and Bos- 
ton College. 



ANAPHORIC 

1. 'Such* functione anaphorical ly mm a deterainer in a noun 
phrase and aay occur in either: 

*euch* ♦* noun 

(4) Tests usre adainistered to students with I . Q.'s 
bordering on slow Isarnsr. Such studsnts frequently 
presented a problea in placement. 




*such^ adjective ♦ noun 
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(5) At on« txmm or ancthart aoiit students take aither 
SAT or GRE taeta. Such atandard taata ara faared by 
Boat atudanta. 

Thia uaa ia diatinguiahabia froa tha aacond appoaitional 
uaa of •auch» in that 'aa* doaa not follow tha noun. 

2. *Sueh* functiona pronoaxnally in tha ayntax: 

•aa' ♦ *auch« 

(6) Tha Statua of Libarty ia conaidered by many 
iaaigranta to ba tha ayabol of fraadoa. Aa such. it 

Mndatory that tha diaintagrating structure be 
reatored. 

DEPENDENT ON TEXT 

»Such» aay aarva aa a pradaterainar for an indefinite noun 
phraaa in tha ayntax: 

*auch* ♦ »a»/»an» ♦ noun 

<7) Syatea ana lye ta recoaaandad a completely neu 
approach to acheduling deliveriea. Such an approach 
uould require extenaive groundwork prior to implemen- 
tation. 

Whether the term *auch* ia functioning anaphorically or not 
dependa on whether the noun in the phraae haa been apecified in 
eny greater detail earlier in text. 
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* Subject* 



Thmtm mtm four abbr^viatMi iorum of rmtmtmnQm to * Subject* or 
'Subjects* in abstracts* naaaly •S** *8*a*» *8a*» or *Sa**. Thaaa 
four posaibla raaliaation foraa oara analyaad aa a aingla group* 
mitb tha following auaaary raaulta. Abbraviatad aubjact rafar- 
anoa occur rad in tba 300 abatracta froa tha PaychAba Oatabaaa* a 
total of 213 tiaaa with 188 of thaaa ooouraneaa baing anaphoric 
for an F. K of •88« Thara uara no ooourrancaa of any of thaaa 4 
abbraviationa in tha 300 abatracta froa INSPEC. 

Of tha 25 non-anaphoric uaaa of tha Subjact abbraviationa* 17 
ara idantif iabla by aatching againat 3 poaaibla contaxtual pat- 
tarna. Tha raaaining 8 occurrancaa ara auoh aora difficult to tag 
aa non-*anaphoric bacaxiaa thair syntactic anvironaanta ara onaa in 
tihicb tha aaaa tara aay ba twad anaphorically. Aa a raault* it 
will ba nacasaary to firat identify all conaiatant anaphoric and 
non-anaphoric pattarna of uaa and than turn to aaaantic analyaia 
to dacida tha atatua of a tara occurring in a pattarn which can 
ba aithar anaphoric or non-anaphoric* 

Tha auggaatad ordar of pattarn-aatching will ba an intar- 
laaving of anaphoric and non-anaphoric rulaa* rathar than firat 
applying all rulaa of ona uaaga in a aaquanca foliouad by all 
rulaa of tha othar uaaga. Tha aoat dafinita* aaaily aatchad pat- 
tarna will ba appliad firat* with thoaa raquiring aora coaplax 
aaaantic procaaaing baing appliad laat. 

1 . Poaaaaa i va - whanavar tha two poaaaac i va f o r aa ara 
obaarvad* thay ara anaphoric. 

S*a/Sa* ^ noun 

(la) 112 collaga atudanta atudiad diffarant aata of 16 
facaa on 3 occaaiona. Analyaia of Sa» conaiatancy ahouad 
that aora than 50% of thaa parforaad conaiatant ly. 

(lb) 112 collaga atudanta atudiad diffarant aata of 16 
facaa on 3 oooaaiona. Analyaia of 112 collaaa atudanta* 
conaiatancy ahowad that aora than 50% of thaa parforaad 
conaiatantly* 



2. Indafinita quantifiar - whan taraa of thia claaa (a. g. 
*aach* * *all** * fawar* * *aoaa* * ate. ) praaodify S/Sa* tha 
S-fora uaa aluaya anaphoric s 

indafinita quantifiar ^ S/Sa 

(2a) Invaatigatad influanoa of 48 houra of alaap dapriva- 
tion (SO) in 12 19-30 yaar old aalaa. Following SO* all Sa 
ahowad aarkad raductiona of DMA aynthaaia. 

(2b) Invaatigatad influanca of 48 houra of alaap dapriva- 
tion (SO) in 12 19-30 yaar old aalaa. Following SO* all 12 
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1 9-30 vf«r old Ml— shousd aarkad reductions o£ DMA 
synthMis. 



3. Initial introduction of subjacta undar study is always non- 
anaphoric and usually of ths fora< 

• ^ trngmi ^ <adjsctivs} S/Ss 

uith sithsr ags or dsseriptivs adjactivs optional » but at 
Isast ons aust bs prsssnt* 

(3a) 8 10 ysar old fssals Ss 

or 

<3b) 8 fssals Ss 
or 

(3c) 8 10 ysar old Ss 



4. Anothsr possibls pat tarn for intrcducing aubjacta* which 
again is non-anaphorict iss 

S/Ss -t* *wsrs* ^ dsscription 

(4) Ss wrs nursing boss rasidsnts with at laast 1 ysar«s 
rssidsncy. 



5* k furthsr non-anaphoric initial introductory pattarn is> 

S/Ss ♦ dsscription 
(5) S Ss> aasd IB to 2i# wars adainistsrsd ths tsst. 



8. Vhsn S/Ss is prsaodifiad by a dsfinits articls or dstsrain- 
ar (s.g* *ths*» *thsss*) ths uss is anaphoric. 

dstsrsinsr ^ S/Ss 

(8a) Adainistsrsd ths Block Dssign subtsst of tha VISC to 
550 assbsrs of 95 aonozygotic twin kinships. Fingsrprint 
ridgs counts of ths Ss wars also analyzsd. 

(6b) Adsinistsrsd ths Block Dssign subtsst of ths VISC to 
550 sssbsrs of 85 sonoxygotic twin kinships. Fingsrprint 
ridgs counts of ths B5Q ssabsrs of BS ■onottvootie kinships 
wsrs slso analyzsd. 

7* Having idsntifxsd ths abovs syntactic snvironssnts* it 
appssrs that ths rsaaining occurrsncss of S/Ss in ths fol- 
lowing contsxt will slwsys indicats anaphoric ussi 
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S/Sm active verb 

C7a) ExparxBMt 1 eosparad racall following aaaantic ori- 
anting inatructiona* foraal orianting inatructiona* and 
intantional iMrning inatructiona uaing 19 undargraduata 
novica ehaaa playara* Sa coanlatad thm Spatial Viatialisa- 
tion Subtaat. 

(7b> Expariaant 1 coaparad racall following aaaantic ori- 
anting inatructiona* foraal orianting inatructiona* and 
intantional laarning inatructiona uaing 19 undargraduata 
novica chaaa playara. 19 undargraduata novica ehaaa nlav- 
9£ft coaplatad tha Spatial Viaualiaation Subtaat* 

8« A fairly coaaon ayntax for *S* to occur in* imt 

adjactiva ♦ S/Sa 

which could ba aithar anaphoric or non-anaphoric dapanding 
on whathar tha *S* had baan apacifiad foraarly. In tha 
graatar proportion of caaaa* tha S-fora ia anaphoric* but 
it ia poaaibla for tha S to ba rafarring rathar abatractly 
and gararally to aubjacta without thair having baan apaci- 
fiad aarliar. 

9. Tha raaaining pattarna of uaa for S/Sa .ara too aingular to 
parait ganaralizad rula-writing* Tharafora* if an occur- 
ranca of 5/Sa doaa not aatch any of tha abova ayntactic 
pattarna* aiaply chack prior taxt to aaa if tha ta^^a haa 
baan apacifiad aarliar. 
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Test Results of Rule Sets 
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Appendix 0^1 

RESULTS OF TESTS OF HOLE SETS 

All rules were tested by at least three people* 
Tester 1 vas involved in the project throughout the first 
year. Tester 2 was not involved except for xrule tests* 
The third and subsequent testers were chosen haphazardly 
from among students in the School of Information Studies 
at Syracuse University* The only requirements were that 
they be native speakers of American English and had not 
previously tested any other rule sets* 

In general, rules were tested by only three people* 
Whenever one of more did not achieve 90% accuracy, or nearly 
so, additional people %rere chosen to test the rules* 
Exceptions to this practice were ^en most of the problems 
arose from rules dealing with ^Aether a concept had been 
specified earlier in greater detail, e*g* "each*, or when 
the number of exanples was so small that one error would 
drop percentages drauaatically* 
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Appendix D-2 



RESULTS OF TESTS OF 
ROLE SETS 



iteaphor 


Number 

Axampxe 

Tested 


Jki^ 

1 


2 


3 


4 


5 


uvercixx 


% Error 
wause<i BY 
2 Rules* 


all 


48 


.792 


.875 


.812 






.879 


37.9 


another 


9 


.778 


.556 


.889 






.741 


85.7 


any 


9 


1.000 


1.000 


.889 






.963 


0.0 


both 


35 


1.000 


.970 


1.000 






.990 


0.0 


do 


50 


1.000 


.980 


.980 






.987 


0.0 


ea^ 


74 


.889 


.824 


.849 


.864 




.857 


92.8 


either 


21 


1.000 


1.000 


1.000 






1.000 


0.0 


it 


58 


.966 


.966 


.966 






.966 


0.0 


less 


35 


.914 


.857 


.909 






.893 


27.3 


many 


26 


.846 


.769 


.731 






.782 


82.4 


Bore 


51 


.843 


.857 


.765 


.608 




.767 


17.0 


most 


35 


.886 


.914 


.914 






.905 


85.7 


much 


10 


.900 


.960 


.700 


.889 




.846 


0-0 


neither 


3 


1.000 


.667 


1.000 






.889 


0.0 


no 


38 


.921 


.921 


.947 






.930 


87.5 


none 


3 


1.000 


1.000 


1.000 






1.000 


0.0 


one 


44 


.977 


.886 


.977 


.795 


.866 


.900 


50.0 


ordinals 


33 


.848 


.848 


.939 






.879 


58.3 


other 


44 


.864 


.907 


.837 






.869 


76.5 


others 


16 


1.000 


.812 


.875 






.896 


0.0 


sane 


38 


.921 


.842 


.789 






.851 


82.4 


so 


11 


1.000 


1.000 


1.000 






1.000 


0.0 


some 


35 


1.000 


1.000 


.971 






.981 


0.0 


such 


61 


.869 


.918 


1.000 






.929 


38.5 


this, that, 
these, thez 


e 42 


.881 


.810 


.333 


.857 




.845 


CO 


vhich 


55 


.982 


.927 


.927 


.818 


1.000 j 


.932 


0.0 



*The t%fo rules causing consistent problems dealt with deciding whether 
a concept was specified in greater detail earlier in text* 
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APPENDIX E 



Retrieval Tests Results 



INSPEC Series ICO 
PsycABS Series 200 
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Page 

A Statlstirai Coaparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 



CENTRAL PRONOUNS 



Correlation Coefficients Significance Level 



0 


s 


TH r. 


r 




z 








JU 




ur 




101 


I 


a 


-•.42A3S7 


-0. 422512 


0.999964 


-1. 198307 


101 


1 


d 


-0. 


-0.753857 


0.979255 


0.320029 


101 


I 


• 


-0.338200 


-0. 376721 


0.938669 


0.600638 


101 


1 


h 


-0.084&90 


-O. 080172 


O. 999317 


-0.62SS87 


101 




J 


-0- 60537^ 


-O. 522589 


0.950945 


-1.581806 


101 




«r> 


-0. 742554 


-0. 731883 


0.969913 


-O. 330174 


101 




n 


-0.6%001 


-O. 694001 


0.969060 


0.057258 


101 


a 


a 


-0. 381796 


-0. 379436 


0.999941 


-1. 188190 


101 




b 


-0.328840 


-O. 327643 


O. 999988 


-1.289199 


101 




e 


-0- 381283 


-0. 379214 


O. 999955 


-1. 185281 


101 






-0. 701048 


-0.698247 


O. 976141 


-0.092007 


101 






-0.339941 


-0.375625 


O. 926487 


O. 508841 


i01 




•f 


-0. 6S2621 


-O. 655125 


0. 9^3046 


-0. 219739 


101 




g 


-0. 701503 


-0. 698217 


0.975441 


-0. 106454 


101 






-0. 031633 


-0.031112 


0.999995 


-O. 832977 


101 


z 


J 


-0.712239 


-0. 697572 


0. 970971 


-O. 438400 


101 


2: 


1 


-0. 001532 


-0. 001532 


1.000000 


O.OOO000 


101 


2. 


M 


-0.731639 


-0. 722978 


0. 973538 


-0.281127 


101 




ri 


-0. 741866 


-0. 750436 


0. 976085 


0.301526 



p > .05 



NOTES : • 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Siailarity Measure: #1 * Cosine. #2 ■ Dice 
TU: Ter« Ueighting Schenes: See Result Page R>1 

correlation Coefficients: rju Is between the user's relevance judgnent and the 
system's predicted relevance based on unresolved anaphors. rjr is between 
the user's relevance Judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's jud^pents were scaled fron low to high (1 « nost relevant. 
4 » Hst non-relevant) a strong negative correlation shows agrecoent 
between user's and system's relevance judgaents. 

Significance Level: A positive Z indicates thit the second correlation is higher 
than the first correlation (rjr> rju). If this Z is statistically 
significant ^s indicated by tne asterisks, then resolving anaphors improves 
the systcn's predications of relevance. 

145 



Appenaix t<»^ 



Page 

A Statistical Coaparlson of the Relationship Between 
Unresolved Anephors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



CENTRAL PRONOUNS 



Correlation Coefficients Significance Level 
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P > .05 



NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: SiMllarlty Measure: II * Cosine. #2 « Dice 
TW: Teni Weighting Scheaes: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance Judgment and the 
systcn's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance Judgaent and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgnents were scaled fro« low to high (1 • nost relevant, 
4 • lost non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rjr> rju). If this Z Is sUtlstlcally 
significant as Indicated ly tne asterisks, then resolving anaphors Improves 
the Siystcm's predications of relevance. 
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A Statistical Conparlson of the Relt: :ish1p Between 
Unresolved Anaphors and User's Relevance x^:znents with Resolved 
Anaphors and User's Relevance Judgnents: Anaphoric Class 
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Correlation Coefficient: 
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Significance Level 



P > .05 



NOTES : 

Q: Queries 100-199 were searched on INSPEC: 20:-J4 on PsychlNFO 
S: Similarity Heasure: #1 > Cosine. 12 * I :: 
TU: Tern Weighting Schemes: See Result Page R-. 

Correlation Coefficients: rju Is between the usr : -elevance judgment and the 
system's predicted relevance based on unres: anaphors. 



the user's relevance judgment and the syste' 
on resolved anaphors. 



rjr Is between 
•■edlcted relevance based 



Because the user's judgments were scaled frr- > to high (1 « most relevant, 
4 « most non-relevant) a strong negative cp-r ::ion stows agreement 
between user's and system's relevance judgn^-: 

Significance Level: A positive Z Indicates that v: :econd correlation is higher 
than the first correlation (rwy rjy). V ■ :l is statistically 
significant as Indicated by tnc istcrlsks, r-z- 'esolving anaphors improves 
the system's predications of relevance. 
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A Statistical Conparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



CENTRAL PRONOUNS 



Correlation Coefficients 
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Q 


s 


107 


1 


107 


1 


107 


1 


107 


1 


J 07 




107 




107 




107 


2 


107 


2 


207 


a 


107 


a 


107 


a 


107 


a 


107 


a 


107 


a 


107 


a 


107 


a 


107 


a 


107 


a 



TH 

• -0. 
d -0. 

• -0. 
h 0. 

J -0. 
i» -0. 
n -0. 
a -0. 
b -0. 
c -0. 
d -0. 

• -0. 
f -0. 
g -0. 
h 0. 

J -0. 
1 0. 
m -0. 
n -0. 



869170 
361714 
001035 
1S8S23 
185056 
327507 
283332 
290729 
285007 
292260 
356617 
001295 
357'f6l 
356079 
182394 
089937 
074756 
315778 
296:99 



-0. 269170 
~0. 343713 
-9. 001095 

0. 152646 
-0. 197007 
-0. 324536 
-0. 289027 
-0. 290729 
-0. 285007 
-0. 292260 
>0. 342283 
-0. 001095 
-0. 343860 
~0. 341662 

0. 181773 
-0. 099660 
0. 074224 
-0. 314926 
-0. 302537 



r 


2 


Ur 




1.O00000 


v« www 


e. 991963 




i.8Me00 


v« www 


0.999996 


"•0. 252577 


0. 996773 


0.624174 


e. 993124 


-0. 103046 


0. 997671 


0.321043 


1.000000 


0. 000000 


1.000000 


0. 000000 


1. 000000 


0. 000000 


0.990676 


-0. 461051 


1.OOOO0O 


0. 000000 


O. 992538 


-0.489780 


0. 990671 


-0. 463463 


O. 999988 


0. 530618 


O. 998077 


0. 649207 


0. 999998 


1.003323 


0. 993906 


-0. 033509 


0. 997877 


0. 419752 



P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 * Cosine. #2 * Dice 
TK: Term Weighting Schemes: See Result Page R-1 

correlation Cbefficients: rju is between the user's relevance judgment and the 
system s predicted relevance based on unresolved anaphors. rjr is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 « most relevant. 
4 ■ most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance jud^nts. 

Significance Level : A positive 2 indicates that the second correlation is higher 
than the first correlation (rir> rjy). If this 2 is statistically 
significant as indicattd by the asterisks, then resolving anaphors improves 
the system's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



CENTRAL PRONOUNS 



Correlation Coefficients 
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Significance Level 



P > .05 



NOTES : 

Q: Queries 100>199 were searched on INSPEC: 200-299 on PsychlNFO 

S: Similarity Measure: II « Cosine. #2 « Dice 

TW: Term Weighting Schemes: See Result Page R-1 ' 

Correlation Coefficients: rjy Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance Judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's Judgments were scaled from low to high (1 « most relevant, 
4 • most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance Judgments. 

Significance Level: A positive Z Indicates that the second correlation is higher 
than. the first correlation (rir> rjy). If this Z is statistically 
significant as Indicated fay tne asterisks, then resolving anaphors improves 
the system's predications of relevance. . . ^ 
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A Statistical Coaparlson of the Relatlonstip Between 
Unresolved Anaphors and User's Relevance Judgnevts with Resolved 
Anaphors and User's Relevance Judgments: for Atisroric Class 
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Correlation Coefficients Significance Level 
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P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 or :s>':nINFO 
S: Similarity Measure: II « Cosine. #2 ■ Dice 
TU: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju is between the user's rt'fu'zt judgnent and the 
system's predicted relevance based on unresolved anrnc-s. rjr is between 
the user's relevance Judgment and the system's prcrr>: relevance based 
on resolved aniphors. 

Because the user's Judgments were scaled from low t: (1 most relevant, 
4 • most non-relevant) a strong negative correlatic^ st'^s agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z indicates that the secorc -rrelatlon Is higher 
than the first correlation (r4,.> rju). If this 2 s j-jtlstically 
significant as Indicated by tne asterisks, then res:'*-^ anaphors Improves 
the system's predications of relevance. ^ 



A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgaents with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



CENTRAL PRONOUNS 



Correlation Coefficients 
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NOTES ; 

Q: Queries 100-199 Mere searched on INSPEC: 200-299 on PsychlNFO 
S: SiMllarlty Heasure: II « Cosine. #2 * Dice 
TW: Tein Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rj^ Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 > most relevant, 
4 ■ most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (r<r> rjy). If this Z Is statistically 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 
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Correlation Coefficients 
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P > .05 
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1.427362 
1 . 664964 
1 . 660726 
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1. 616615 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 

S: Similarity Measure: #1 * Cosine. #2 « Dice 

TW: Term Weighting Schemes: See Result Page R-1 * 

Correlation Coefficients: rj^ Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rw Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's Judgments were scaled from low to high (1 « most relevant, 
4 ■ most non-relsvant) a strong negative correlation shows agreement 
between user's and system's relevance Judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation {.tw^ rjy). If this Z Is statistically 
significant as Indicated by tne asterisks, then resolving anaphors improves 



th« lystam's predications of relevance. 
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Unresolved Anaphors and User's Relevance Judgnents with Resolved 
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CENTRAL PRONOUNS 



Correlation Coefficients 
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Significance Level 



P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 

S: Similarity Measure: #1 « Cosine. 12 * Dice 

TH: Term Weighting Schemes: See Result Page R-1 ' 

Correlation Coefficients: is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments Were scaled from low to high (1 « most relevant, 
4 ■ most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation Kr^y rju). If this Z is statistically 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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A Statistical Coaparlson of tte Relationship Between 
Unresolved Anaphors and User's Re1e*ince Judgnents with Resolved 
Anaphors and User's Relevance Judgir^nts: for Anaphoric Class 



CENTRAL PRONOUN! 



Correlation Coeff':1ents Significance level 
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P > .05 



NOTES: 

Q: Queries 100-19S were searched on IH^.li 200-299 on PsychlNFO 
S: Similarity Measure: #1 > Cosine. «2 « Dice 
TW: Term Weighting Schemes: See Result ^ice R-1 

Correlation Coefficients: rju Is betweer -jie user's relevance judgment and the 
system's predicted relevance based ot anresolved anaphors. rjr Is between 
the user's relevance judgment and thi system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were sci ^ from low to high (1 « most relevant, 
4 « most non-relevant) a strong nege: /« correlation shows agreement 
between user's and system's re1evanc< jjdgments. 



Significance Level: A positive Z Indicate: -..nat the second correlation Is higher 

If this Z Is statistically 
us, then resolving anaphors Improves 



than the first correlation (rjr> r^^ If this Z Is statistically 
significant as Indicated by the aste-i«s, t 
the liystem's predications of relevan:^. 
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A Statistical Conparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgaents: for Anaphoric Class 



CENTRAL PRONOUNS 



Correlation Coefficients Significance Level 
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NOTES: 

Q: luerles 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: 41 * Cosine, #2 « Dice 
TW: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rj^ Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rjr is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's Judgments were scaled from low to high (1 « most relevant* 
4 « most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance Judgments. 

Significance Level: A positive Z Indicates that the second correlation is higher 
than the first correlation (rir> rjy). If this Z Is statistically 
significant as indicated by tne asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



CENTRAL PRONOUNS 



Correlation Coefficients 



Significance level 
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P > .05 



NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 

S: Similarity Measure: II * Cosine. #2 « Dice 

TW: Term Weighting Schemes: See Result Page R-1 ' 

Correlation Coefficients: rju Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
en resolved anaphors. 

Because the user's Judgments were scaled from low to high (1 • most relevant, 
4 • most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance Judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rir> rjy). If this Z Is statistically 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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A Statistical Coaparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 

NOMINAL DEMONSTRATIVES 



Correlation Coefficients 



Significance Level 
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p > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 

S: Similarity Measure: #1 > Cosine. #2 « Dice 

TU: Tern Weighting Schemes: See Result Page R-I ' 

Correlation Coefficients: rju Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 * most relevant, 
4 • most non-relevant) • strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation irw> rju). If this Z Is statistically 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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A Statistical Coaparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 



NOMINAL DEHONSTRATIVES 



Correlation Coefficients 



Significance Level 
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NOTES ; 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Slirilarlty Measure: #1 « Cosine. #2 « Dice 
TW: Tem Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 « most relevant, 
4 * most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance Judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (ry^ rju). If this Z is statistically 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



NOMINAL DEMONSTRATIVES 



Correlation Coefficients 



Significance Level 
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NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 

S: Similarity Measure: II « Cosine. 12 « Dice 

TU: Term Weighting Schemes: See Result Page R-1 ' 

Correlation Coefficients: rju Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rw Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 « most relevant, 
4 « most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance Judgments. 

Significance Level : A positive Z Indicates that the second correlation is higher 
than the first correlation (r*r> rju). If this Z Is statistically 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 

NOMINAL DEMONSTRATIVES 



Correlation Coefficients 



Significance Level 
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1. 161495 


0. 997489 


1.350585 


0. 999973 


1.221262 


0. 997207 


0. 822671 


0. 996274 


1.078252 



P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 « Cosine. 12 « Dice 
TH: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance Judgment and the 
system's predicted relelrance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's Judgments were scaled from low to high (1 » most relevant, 
4 • most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance Judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation {rw> rju). If this Z is statistically 
significant as Indicated tne asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



NOMINAL DEMONSTRATIVES 



Correlation Coefficients 



Significance Level 
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NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 

S: Similarity Measure: #1 « Cosine. #2 « Dice 

TW: Term Weighting Schemes: See Result Page R-1 ' 

Correlation Coefficients: rju Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 « most relevant, 
4 « most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 



Significance Level: A positive Z Indicates that the second correlation Is higher 

' ' "f this Z is sutlstically 

then resolving anaphors Improves 



than the first correlation (r4r> rjy). If this Z Is sutlstically 
significant as Indicated by tne asterisks, t 



the system's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgm«ints: for Anaphoric Class 



NOMINAL DEMONSTRATIVES 



Correlation Coefficients 



Significance Level 



TW 



ur 



P > .05 
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1.704237 
1.049544 
0. 000000 
0. 326041 
0. 390669 
1.014222 

-1.067418 
1.390371 
1.247222 
1.418114 
0. 831461 
0. 000000 
0. 538514 
0. 805419 
0. 755600 
0. 739402 
0. 465540 
0. 776941 

-1. 195659 



NOTES: 

Q; Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 

S: Similarity Measure; #1 « Cosine. #2 « Dice 

TH; Term Weighting Schemes: See Result Page R-1 ' 

Correlation Coefficients: rjy 1$ between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rw is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 > most relevant, 
4 « most non-relevant) a strong negative correlation shows agreanent 
between user's and system's relevance judgments. 

Significance Level: A positive Z indicates that the second correlation is higher 
than the first correlation (r4,.> rj^). If this Z 1$ statistically 
significant as indicated by the asterisks, then resolving anaphors improves 
the system's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



NOMINAL DEMONSTRATIVES 



Correlation Coefficients 



Significance Level 
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-0. 339456 
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i. 000000 


0. 000000 
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-0. 352275 


-0.412972 


0. 96£1£6 
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-0. 764884 
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-0. 00£870 
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0. 990437 
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fi 


-0.254817 


-0. £73608 


0. 995269 


0. 871430 



P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 

S: Similarity Measure: #1 « Cosine. 12 * Dice 

TW: Term Weighting Schemes: See Result Page R-1 * 

correlation Coefficients: rju Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment a?id the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 « most relevant, 
4 « most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation iryy rjy). If this Z Is statistically 
significant as Indicated fey tne asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
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NOMINAL DEMONSTRATIVES 



Correlation Coefficients Significance Level 
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-0. 16£91 1 


-0. 189500 


0. 991659 


0. 781175 
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a 
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0. 285531 
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0.000000 
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0.0163! 4 
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0. 983367 
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0. 997716 


0. 952612 
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2. 
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-0. 103790 
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0. 994449 


1.306359 


158 
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0. 999693 


1.006950 
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-0. 131206 


-0. 181133 


0. 990884 


1.394890 
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-0. 120778 


-0. 138402 


0. 994413 


0. 628804 



P > .05 



NOTES: 

Q: Quer1i>s 100-199 were searched on INSPEC: 200-299 on PsychlNFO 

% 

S: Similarity Measure: II = Cosine. 12 « Dice 

TU: Term Weighting Schemes: See Result Page R-1 ' 

Correlation Coefficients: rjy Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rw Is between 
the user's relevance Judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's Judgments were scaled from low tc high (1 • nost relevant, 
4 « most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance Judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation {t\t> ''Ju)* ^^^^ ^ ^< statistically 
significant as Indicated by tne asterisks, then resolving aratphors Improves 
the system's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
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NOMINAL DEMONSTRATIVES 



Correlation Coefficients Significance Level 
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NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 

S: Similarity Measure: #1 " Cosine. 12 « Dice 

TW: Term Weighting Schemes: See Result Page R-1 ' 

Correlation Coefficients: rjy Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rw Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 « most relevant, 
4 ■ most non-relevant) a strong negative correlation sho^ agreement 
between user's and system's relevance Judgments. 

Significance level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (r<p> rjy). If this Z Is statistically 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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NOMINAL DEMONSTRATIVES 



Co rrelatlon Coefficients 



Significance Level 
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P > .05 
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NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 

S: Similarity Measure: II * Cosine. 12 ' Dice 

TW: Term Weighting Schemes: See Result Page R-1 * 

Correlation Coefficients: rjy Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 > most relevant, 
4 « most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (ri^ > ''ju)* ^^^^ ^ Is statistically 
significant as Indicated by tne asterisks, then resolving anaphors in^roves 
the system's predications of relevance. 
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Correlation Coefficients Significance level 
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0. 954375 
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p > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 

S: Similarity Measure: #1 « Cosine. #2 « Dice 

TW: Term Weighting Schemes: See Result Page R-1 * 

Correlation Coefficients: rju Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 « most relevant, 
4 « most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z indicates that the second correlation Is higher 
than the first correlation {rw> rju). If this Z Is statistically 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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Unresolved Anaphors and User's Relevance Judgments with Resolved 
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NOMINAL DEMONSTRAT IVES 



Correlation Coefficients 



Significance Level 
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-0. 024803 
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NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 

S: Similarity Measure: II » Cosine. 12 « Dice 

TU: Term Weighting Schemes: See Result Page R-1 ' 

Correlation Coefficients: rj^ Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance Judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's Judgments were scaled from low to high (1 « most relevant* 
4 • most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance Judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation {rwy fjy). If this Z is statistically 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



RELATIVE PRONOUNS 



Correlation Coefficients 



Significance Level 
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p > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 « Cosine. 12 ^ Dice 
TW: Term Weighting Schemes: See Result Page R-1 



Correlation Coefficients: 
system's predicted rel 

the user's relevance Judgment and the system's predicted relevance based 
on resolved anaphors. 



rju Is between the user's relevance judgment and the 
evance based on unresolved anaphors. rjr is between 



Because the user's Judgments were scaled from low to high (1 * most relevant* 
4 ■ most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance Judgments. 

Significance Level: A positive Z Indicates that the second correlation is higher 
than the first correlation (rjr> rju)- If this Z is statistically 
significant as indicated by tne asterisks, then resolving anaphors improves 
the system's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



RELATIVE PRONOUNS 



Correlation Coefficients Significance Level 
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NOTES: 

Q: Queries 100-ig9 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 > Cosine. #2 « Dice 
TW: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rw Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 » most relevant, 
4 • most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (r<p> rjy). If this Z Is statistically 
significant as Indicated by the asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



RELATIVE PRONOUNS 



Correlation Coefficients 
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Significance Level 



p > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: §1 • Cosine. #2 « Dice 
TW: Term Weighting Schemes: See Result Page R-1 



Correlation Coefficients: 

system's predicted rele^anct: based on unresolved anaphors. rjr Is between 
the user's re1r(/ance judgment and the system's predicted relevance based 
on resolved anaphors. 



rju Is between the user's relevance judgment and the 
levi 



Because the user's judgments were scaled from low to high (1 « most relevant, 
4 « most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (r4y.> rjy). If this Z Is statistically 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



RELATIVE PRONOUNS 



Correlation Coefficients 



Significance level 
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p > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 » Cosine. #2 « Dice 
TW: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rjy is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 « most relevant, 
4 ■ most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance level: A positive Z Indicates that the second correlation is higher 
than the first correlation (r4r> rju). If this Z Is statistically 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



RELATIVE PRONOUNS 



Correlation Coefficients 



Significance Level 
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P > .05 



NOTES: 

Q : Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Me«^sure: #1 > Cosine. 12 « Dice 
TW: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance Judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 « most relevant, 
4 « most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance Judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation KT\^y rjy). If this Z Is statistically 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the system's predications of relevance. 

173 



Page 
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RELATIVE PRONOUNS 



Correlation Coefficients 
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Significance Level 



P > .05 
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NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 > Cosine. #2 « Dice 
TW: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 * most relevant. 
4 « most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (r<,.> rjg). If this Z Is statistically 
significant as Indicated by tne asterisks t then resolving anaphors 1iiv)roves 
the system's predications of relevance. 
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RELATIVE PRONOUNS 



Correlation Coefficients 
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p > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Ne£$ure: #1 * Cosine. #2 « Dice 
TW: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rjr is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Secause the user's Judgments were scaled from low to high (1 > most relevant, 
4 « most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance Judgments. 

Significance Level: A positive Z Indicates that the second correlation is higher 
than the first correlation [rwy rjy). If this Z is statistically 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



RELATIVE PRONOUNS 



Correlation Coefficients 



Significance Level 
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p > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: II » Cosine. #2 « Dice 
TW: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 > most relevant, 
4 » most non-relevant) a strong negative correlation shows agreement 
between user's and system^s relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rw> rjy). If this Z Is statistically 
significant as Indicated by tne asterisks, then resolving anaphors improves 
the system's predications of relevance. 
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Page 

Comparison of the Relationship Between 

and User's Relevance Judgments with Resolved 

Relevance Judgments: for Anaphoric Class 



RELATIVE PRONOUNS 



Correlation Coefficients 



Significance Level 





c 


TW r. 


1^ 




7 












ur 




170 


1 


• 


-0. 716&13 


-0. 716£01 


0. 999987 


-0.500158 


i70 


1 


a 


—0. 7^1051 


-0. 710474 


0. 987554 


-0. 801££S 


170 


1 




—A 1 1 AO 
"W. V<0i 1V7 


-0. 001109 


1.000000 


0. 000000 


1 70 


1 


n 


A 1 CCC 1 9 


0. 16£578 


0. 999901 


1. £60646 


170 


1 


J 


-0. S9456£ 


-0. 606£e8 


0. 958£30 


0.£££149 


170 


1 


m 


-0. &36£59 


-0. 6£ei39 


0. 984468 


-0. £59331 


170 


1 


n 


-0.515708 


-0. 51310£ 


0. 977975 


-0. 063£36 


170 


£ 


m 


-0. &7B385 


-0. 6781 1£ 


0. 999979 


-0. £49519 


170 


£ 


b 


-0. 6375SB 


-0. 637414 


0. 999996 


-0. £74505 


170 


2 


c 


-0. 678090 


-0. 677830 


0. 999983 


-0. £6£757 


170 


& 


d 


-0.691 708 


-0. 681 £00 


0. 99473£ 


-0. 605948 


170 


& 


• 


-0. 001109 


-0.001109 


1 . 000000 


0. 000000 


170 


£ 


f 


-0. 696939 


-0. 68863£ 


0. 996515 


-0. 593786 


170 


£ 


g 


-0. 689511 


-0. 67e£55 


0.995194 


-0. 675007 


170 


£ 


h 


0. 153761 


0. 1497£0 


0. 9996£1 


0. 646697 


170 


£ 


J 


-0. 553947 


-0. 561 £81 


0. 968401 


0. £5£795 


170 


£ 


1 


0. 045678 


0. 04£893 


0. 999890 


0.817551 


170 


£ 


m 


-0. 6£3943 


-0.618165 


0. 993£3£ 


-0« £75853 


170 


£ 


Yt 


-0. 5337£8 


-0.531134 


0.99001£ 


-0. 09458S 



p > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: II « Cosine. #2 * Dice 
TW: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 * most relevant, 
4 « most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation is higher 
than the first correlation (np> rju). If this Z Is statistically 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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•A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



RELATIVE PRONOUNS 



Correlation Coefficients 



Significance Level 
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NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 « Cosine. #2 « Dice 
TW: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rj^ is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 * most relevant, 
4 • most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z indicates that the second correlation Is higher 
than the first correlation (rir> rju). If this Z is statistically 
significant as Indicated by tne asterisks, then resolving anaphors improves 
the systcai's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User 'l»rRe1evance Judgments: for Anaphoric Class 



RELATIVE PRONOUNS 
Correlation Coefficients Significance Level 
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P > .05 



NOTES: 

Q: Queries 100*199 were searched on INSPECT 200-209 on PsychlNFO 
S: Similarity Measure: #1 « Cosine. #2 « Dice 
TW: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rjy is between the user's relevance judgment and the 
system's predicted relevance bssed on unresolved anaphors. rj^ Is between 
the user's relevance Judgment and the system's predicted relevance based 
on resolved anaphors. 

Secau'te the user's judgments were scaled from low to high (1 « most relevant, 
4 « most non-retevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance level: A positive Z Indicates that the second correlation Is higher 
then the first correlation (r<p> rju). If this Z 1$ statistically 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
rr»Y/- ths system's predications of relevance. 

179 



A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



RELATIVE PRONOUNS 



Correlation Coefficients Significance Level 
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NOTES; 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 * Cosine. #2 « Dice 
TW: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments Mre scaled from low to high (1 « most relevant, 
4 « most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (r<t.> r^^^), If this Z is statistically 
significant as indicated by tne asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 
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Significance Level 



P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Keasure: #1 « Cosine. #2 « Dice 
TW: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: r^^^ is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rj^ is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 « most relevant, 
4 « most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significartce Level: A positive Z indicates that the second correlation is higher 
than the first correlation (r<r> rjy). If this Z is statistically 
significant as indicated by toe asterisks, then resolving anaphors improves 
the system's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



NOMINAL SUBSTITUTES 



Correlation Coefficients 



Signi ficance Level 
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1.000000 


0. 000000 


0. 999996 


0. 4aSl98 


0. 999987 


0. 391681 


0. 999963 


0.718655 


0. 998460 


0. 551479 


0. 999992 


-0. 220230 


0. 999843 


0. 041967 


0. 999793 


0. 157330 



p > .05 



NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200>299 on PsychlNFO 
S: Similarity Measure: #1 « Cosine. 12 « Dice 
TW: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju 1$ between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 « most relevant, 
4 « most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (r*^^ rjy). If this Z Is statistically 
significant as Indicated by the asterisks, then resolving anaphors iinproves 
the system's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



NOMINAL SUBSTITUTES 



Correlation Coefficients 



Significance Level 



10A 
10A 
10A 
10A 

104 
104 
104 
104 
104 
104 
104 
104 
104 
104 
104 
104 
104 
104 
104 



a 

& 
& 

& 

Z 
2 



TW 

• -0. 

d -0. 

• ~0. 

h -0. 

J -e. 

m -0. 
tt -0, 
a -0. 
b -0. 

c -0. 

d -0. 

• -0. 

f -0. 

S -0. 

h -0. 

J -0. 
1 -0, 
m -0, 
-0. 



071 £62: 
£23340 
4S9965 
347650 
470712 
602159 
415472 
153961 
161980 
149594 
592005 
426036 
577165 
595353 
178700 
472299 
017455 
565612 
384929 



jr 

-0.071204 
-0.619571 
-0.421939 
-0. 337124 
-0. 447830 
-0. 595621 
-0. 368309 
-0. 154278 
-0. 162270 
-0. 149793 
-0. 566358 
-0. 409160 
-0. 537765 
-0. 566049 
-0. 147296 

-0. 453364 
-0.014014 
-0. 543397 
-0. 342904 



ur 

0. 999904 
0. 997626 
0. 998045 
0. 994212 
0. 993779 
0. 99650? 
0. 986257 
0. 999659 
0. 999969 
0. 999884 
0. 996800 
0.991247 
0. 994746 
0. 996264 
0. 995153 
0. 995a&& 
0. 999954 
0. 996573 
0. 991613 



p > .05 



-0. 018776 
-0. 311435 
-0. 631066 
-03 464893 
-1.016407 
-0. 434985 
-1.355409 
0. 060141 
0. 165467 
0. 059193 
-;t. 637171 
-0. 624896 
-1,892411 
-1. 723080 
-1. 440278 

-1.032890 
-1.610789 
-1.378252 
-1. 522880 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: II * Cosine. 12 * Dice 
TU: Term Weighting Schetnes: See Result Page R-1 

Correlation Coefficients: rju 1$ between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr is between 
the user's relevance Judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's Judgments were scaled from low to hioh {1 * most relevant, 
4 * most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance Judgments. 

Significance Level: A positive 2 indicates that the second correlation is higher 
than the first correlation (r4r> rjy). If this Z is statistically 
significant as indicated by tne asterisks, then resolving anaphors improves 
the system's predications of relevance. 
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A Stdtlstlcal Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



NOMINAL SUBSTITUTES 



Correlation Coefficients Significance Level 



Q 


s 


TW r. 




r 


Z 










•^jr 


u r 






1 




—A PAQ 1 701 




\m 0000^0 


0. 000000 


1 Qi7 


t 


H 
w 








-0. 901 149 


107 






— d. 001(995 




1 • 000000 


0. 000000 


107 


I 


h 


0. 15S533 






—0. d784o0 




1 


J 








— 0. o4576«^ 


at!?/ 


X 




-0 327307 


«51 /o 




-~0» 610309 


Xmff 


1 


r 1 












o 


a 


-0. £907£9 


—A OQA^OQ 




0« 000000 


107 


£ 


b 


-0. £65007 


-0. ir85007 


1.000000 


0. 000000 


107 




c 


-0. £9££60 


-0. £9££60 


1 • 000000 


0. 000000 


107 


£ 


d 


-0. 356617 


-0. 3407£3 


0. 93668^ 


-0. 878567 


107 




• 


-0. 001035 


-0.001095 


1.000000 


0. 000000 


107 


£ 


f 


-0. 357481 


-0. S40£85 


e. 996814 


-0. 938955 


107 


£ 


g 


-0. 356079 


-0. 339461 


0. 99S661 


-0.887019 


107 


£ 


h 


0. ia£394 


0. 181934 


0. 999986 


0. 394991 


107 


£ 


J 


-0. 089937 


-0. 085541 


0. 99938£ 


-0.517402 


107 


£ 


1 


0. 074756 


0. 074394 


0. 999998 


0. 696S53 


107 


£ 


m 


-0. 315778 


-0. 306067 


0e 997936 


-0. 661291 


107 


£ 


n 


-0. £96199 


-0. £9£580 


0. 999205 


-0. 390966 



P > .05 



NOTES: 

Q: Queries 100«199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: II * Cosine. #2 * Dice 
TW: Term Weighting Schemes; See Result Page R-1 

correlation Coefficie'-ts: rju is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr is between 
the user's re1evi:nce judgment and the system's predicted relevance based 
on resolved anapf-^ors. 

Because the user's judgments were scaled from low to high (1 * most relevant, 
4 « most non>relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation is higher 
than the first correlation (n^> rjy). If this Z is statistically 
significant as Indicated by tne asterisks, then resolving :.naphors Improves 
the system's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



NOMINAL SUBSTITUTES 



Correlation Coefficients 



Significance Level 



TW 



ur 



P > .05 



1 0^ 










1 • 000000 


0. 000000 


103 




d 


%rf "-v 






!• ^66514 


103 




IP 


m w 






0, 000000 


103 




h 










103 






-0. 1314&2 








103 




rn 


-0. 328732 


-0. 328888 


0. 996127 


0.0100&3 


109 




n 


-0. 356384 


-0. 353835 


0. 992131 


-0. 206230 


103 




a 


-0.228134 


-0.228134 


1 . 000000 


0. 000000 


103 


£ 


b 


-0. 233S10 


-0. £33510 


1. 000000 


0. 000000 


103 




c 


-0. 234533 


-0. 234593 


1 . 000000 


0. 000000 


103 


2 


d 


-0. 285177 


-0. 298763 


0. 997576 


1.093045 


103 


s. 


m 


-0. 000545 


-0. 000545 


1 . 000000 


0. 000000 


103 


z 


f 


-0. £93837 


-0. 303923 


0.998215 


0. 948744 


103 




g 


-0. £895&£ 


-0. 302522 


0. 997656 


1.061412 


103 


2 


h 


-0. 088850 


-0. 069777 


0. 939947 


0. 483698 


103 


2 


J 


-0. 1253,51 


-0. 120007 


0. 933550 


-1.073770 


103 


2 


1 


-0. 063214 


-0. 070230 


0. 398914 


0. 124575 


103 


2 


m 


-0. 236651 


-0. 236048 


0. 996755 


-0. 042274 


103 


2 


ri 


-0. 330370 


-0. 322610 


0. 933413 


-0. 384351 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 » Cosine. #2 « Dice 
TH: Term Weighting Schemes: See Result Page R-1 

Correlation coefficients: r*^ U between the user's relevance judgment and the 
system s predicted relevance based on unresolved anaphors. ri> is between 
the user s relevance Judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's Judgments were scaled from low to high (1 « most relevant, 
4 « most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance Judgments. 

Significance Level: A positive Z Indicates that the second correlation is hig»»er 
than the first correlation (r<r> rju). If this Z Is statistically 
significant as Indicated by the asterisks, then resolving anaphors improve* 
the system's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



NOMINAL SUBSTITUTES 



Correlation Coefficients 



Significance Level 



Q 


s 


135 


1 


1 ^Zi 


1 




1 




1 




1 


loo 


1 


135 


1 


135 


2 


135 


2 


135 


Z 


135 


z 


135 


£ 


135 




135 


£ 


135 


£ 


135 


£ 


135 


£ 


135 


£ 


135 


£ 



TW 

a -0 
d -0 
m -0 

h ~0 

J -0. 
m ~0. 
ri -0. 
a -0. 
b -0. 

c -0. 

d -0. 
e -0. 
f -0. 
g -0. 
h -0. 

J -0, 
1 -0. 
m -0. 
ri -0. 



. S90&94 
. 797£36 
.001234 
. 0££503 
63397£ 
796876 
84£13£ 
637630 
647093 
635£40 
816316 
001 £34 
618960 
816166 
001788 
766710 
001 3£9 
813643 
8£6860 



-0. 600904 
-0. 798£53 
-0.001 £34 
-0. 0356£6 
-0. 644£1£ 
-0. 800067 
-0.811076 
-0. 648359 
-0. 65S380 
644584 
-0.819110 
-0.001 £34 
-0.ei89£6 
-0.817176 
-0. fc0£l£0 

-0. 778213 
-a. 001392 
-0.816604 
-0. 797563 



r 


z 


U V 




0. 999613 


1 . 704£37 


0. 999966 


0. 806£27 


1 • 000000 


0. 000000 




0. 80547£ 


0* 9602:61 


0. £76614 


0» 999716 


0. 876£3l 


0. 995357 


-1.8£9803 


Ok OQOOsrn 


1. 390371 


0. 999766 


l.£47££2 


0. 999466 


1.416114 


®. 999765 


0. 261511 


1 . 000000 


0. 000000 


0. 999918 


-0. 019£73 


0. 999821 


C". 37045£ 


0. 999993 


1.315987 


0. 995616 


0. 773603 


1 . 000000 


0. 479305 


0. 999662 


0. 78153£ 


0. 995690 


-1.778178 



P > .05 



NOTES: 

Q: Queries 100-199 were searched on IHSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 « Cosine. #2 * Dice 
TH: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju 1$ between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rw is between 
the user's relevance Judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's Judgments were scaled from low to high (1 * most relevant, 
4 « most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance Judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation {r*^y rjy). If this Z Is statistically 
significant as Indicated by the asterisks, then resolving anaphors 1iiq}roves 
the system's predications of relevance. ^ _ 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



NOMINAL SUBSTITUTES 



Correlation Coefficients S ignificance Level 



Q 


s 


TW r.. 


r . 




7 












ur 




Inc. 








— 0. lo95%52 


0. 99996^ 


— i« 337336 


4 AO 




O 






0. 979»icil 


0. 901^495 


4 A O 










1 • 000000 


0. 000000 


1 Ai> 




h 


-0. 277455 




Ok Q4*^OA7 




1 A5> 










A O 4 0*7A£r 


"-0* l«»Dcflo 


1 

1 He 




Iff 


W» Vt X V r 






771676 






ITi 




ZOVO0H 






1 AO 


c 


A 


-0. 324310 






~1 • 2o5Sb>3 


1A£ 


2 


b 


-e. 435620 


-0. 434607 


0. 999992 


-1.209647 


142 


2 


c 


-0. 355980 


-0. 353568 


0. 999961 


-1.251661 


J4£ 


2 


d 


>0. 339458 


-0. 373569 


0. 976897 


0. 737341 


14£ 


2 




-0. 000662 


-0. 000662 


1 • 000000 


0. 000000 


1A& 


2 


f 


-0. 452453 


-0. 472702 


0. 979268 


0. 487727 


142 


2 


□ 


-0. 352275 


-0. 385374 


0. 976469 


0.712576 


14£ 


2 


h 


-0. 277419 


-0. 054444 


0.216294 


-0. 805042 


14£ 


2 


J 


-0. 316247 


-0. 345299 


0.916169 


0. 328365 


142 


2 


1 


-0. 255325 


-0. 002206 


0. 398154 


-1.037952 


142 


2 


ffl 


-0. 359307 


-0. 348560 


0. 996619 


-0. 943022 


142 


2 


•n 


-0.254817 


-0. 238696 


0. 995236 


-0. 741030 



P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPtC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 « Cosine. #2 » Dice 
TW: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 « most relevant. 
4 * most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z indicates that the second correlation is higher 
than the first correlation (n^ > rju). If this Z is statistically 
significant as indicated by tne asterisks, then resolving anaphors improves 
the system's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 

NOMINAL SUBSTITUTES 



Correlation Coefficients 



Significance Level 



Q 


S 




r . 


r 

ur 


z 


D > 


158 


1 


* 


0.036316 


0. 037274 


0. 999861 


0. 234405 




156 


1 


d 


-0. 162911 


-0. 167073 


0. 999889 


1. 058125 




158 


1 


• 


-0. 0010SS 


-0. 001032 


1.000000 


0. 00000(2) 




156 


1 


h 


-0. 2&1 133 


"Z. ££8^06 


0. 99964.^ 


1 . 034££t:^i:: 




156 


1 


J 


-0. 172847 


-0. 187003 


0. 991 193 


0. 405551 




156 


1 


m 


-0.212637 


-0. 221968 


0. 999772 


1 . 656869 
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1 


n 


-0. 190370 


*0« 210136 


0. 992971 


0. 635701 




156 


& 


a 


0. 090669 


0. 089328 


0. 999609 


0. 257478 
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& 


b 


0. 174475 


0. 173910 


0. 999972 


0. 285531 
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Z 


c 
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115103 


0. 999875 


0. 244216 
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Z 


d 
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0.413479 
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• 
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»0. 001032 
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0. 000000 
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z 


f 
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0.014657 


0. 999962 
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z 
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-0.091100 


0. 999877 


0. 429229 
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z 
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-0. 175037 


0. 997409 


2. 004835 
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z 


3 


-0. 103790 


-0. 120490 


0. 998705 


1 . 233151 
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£ 


1 


-0. 007262 


-0. 020390 


0. 999677 


1« 932696 




156 


z 


m 


-0. 131206 


-0. 139479 


0. 999815 


1.617782 




156 


z 


Vi 


-0. 120778 


-0. 136954 


0. 996159 


0. 695633 





.05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure; #1 « Cosine. 12 " Dice 
TW: Ter 'eighting Schemes: See Result Page R-1 

correlation Coefficients: rju is between the user's relevance judgment and the 
systen's predicted relevance based on unresolved anaphors. rj> is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 ' most relevant, 
4 « Most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation is higher 
than the first correlation (rir> rju). If this Z is statistically 
significant as Indicated by tne asterisks, then resolving anaphors 1 
^ d the system's predications of relevance. 
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Improves 



A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



NOMINAL SUBSTITUTES 



Correlation Coefficients Significance Level 



Q 


s 


TW r . 


r . 




7 








JU 


jr 


ur 




170 






-0.716613 


-0. 716201 


0. 999987 


-0. 500158 


170 




d 


-0. 731051 


-0. 734416 


0. 998726 


0. 423922 
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e 
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-0. 001109 
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h 
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0. 999959 


0. 810368 
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J 


-0. 594562 
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0. 447377 
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rn 


-0. 636259 


-0. 640931 


0. 998552 


0. 487940 


170 




n 


-0. 515706 


-0. 523314 


0. 998513 


0. 704176 


170 




a 


-0. 678385 


-0. 678112 


0. 999979 


-0. 249519 


170 




b 


-0. 637558 


-0. 637414 


0. 999996 


-0. 274505 


170 




c 


-0. 678090 


-0. 677830 


0. 999983 


-0- 262757 


170 


2 


d 


-0.691708 


-0. 696222 


0. 999350 


0. 742064 


170 


2 


« 


-0. 001109 


-0. 001109 


1 . 000000 


0. 000000 


170 


2 


f 


-0. 696939 


-0. 699257 


0. 999797 


0. 687098 


170 


2 


g 


~0. 689511 


-0. 694292 


0. 999377 


0. 798120 


170 


2 


h 


0. 153761 


0. 153183 


0. 999984 


0. 454847 


170 


2 


J 


-0. 5539A7 


-0. 558604 


0. 999150 


0. 587486 


170 


2 


1 


0. 045678 


0. 045931 


0. 999997 


-0. 476418 


170 


2 


m 


-0. 623943 


-0. 628105 


0. 999382 


0. 652716 


170 


2 


rj 


-0. 533728 


-0. 539636 


0. 999363 


0. 841414 



P > .05 



NOTES: 

Q: Queries 100-J99 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: II » Cosine. #2 « Dice 
TW: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 « most relevant. 
4 » most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Slgnificince Level: A positive 2 indicates that the second correlation is higher 
than the first correlation (n,.> rjy). If this Z Is statistically 
significant as indicated by tne asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



NOMINAL SUBSTITUTES 



Correlation Coefficients 



Significance Level 
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b 
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c 
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a 


d 
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2 


m 


-0. 434632 
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-0. 265615 


160 


a 


f 


-0. 393963 


-0.410412 


0. 999072 


2. 002646 


160 


a 


D 


-0. 468597 


-0. 466036 


0. 997586 


1. 390247 


160 


a 


h 


-0. 352302 


-0. 339907 


0. 967043 


-0. 257435 


180 


a 


J 


-0. 449142 


-0. 451712 


0. 996600 


0. 293474 


160 


a 


1 


-0. 126480 


-0. 119456 


0. 996554 


-0. 426230 


180 


a 


m 


-0. 476596 


-0- 482945 


0. 997654 


0. 526265 


160 


a 


tt 


-0.445123 


-0.441612 


0. 996257 


-0. 331234 



p > .05 



NOTES: 

Q: Queries 100>199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 Cosine. #2 * Dice 
TW: Term Weighting Schemes: See Result Page R>1 

Correlation Coefficients: rjy Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rj^ is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 ^ most relevant. 
4 ■ most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 



Significance Level: A positive Z indicates that the second correlation is higher 

■ F this Z 1$ statistically 

then resolving anaphors improves 

190 



than the first correlation (nr> rjy). If this Z Is statistically 
significant as Indicated by the asterisks, X 



the system's predications of relevance. 



A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



NOMINAL SUBSTITUTES 



Correlation Coefficients Significance Level 



Q 


s 


TW r. 


r . 


p 


7 








JU 




ur 




162 




a 


-0. 194413 


-0. 192936 


0. 999984 


-1. 386175 


1B2 


1 


d 


-0.221779 


-0. 205365 


0. 995631 


-0. 949108 


16£ 




e 


-0. 000462 


-0. 000462 


1.000000 


0. 000000 


18£ 




h 


0. 185728 


0. 189: 37 


0. 999911 


-1.475952 


laE 


1 


J 


~0. 008062 


0.023412 


0. 994536 


-1.593812 


IBS 


1 


m 


-0. 187365 


-0. 149671 


0. 990446 


-1.460139 


182 


1 


ri 


-0. 166005 


-0. 120033 


0.987143 


-1. 530116 


182 


2 


a 


-0. 174304 


-0. 172579 


0. 999979 


-1. 414123 


182 


2 


b 


-0. 101067 


-0. 100631 


0. 999999 


-1. 424118 


182 


2 


c 


-0. 137425 


-0. 136297 


0. 999991 


-1.417226 


162 


2 


d 


-0. 223362 


-0. 214499 


0. 997090 


-0. 629463 


182 


2 


c 


-0. 037327 


-0. 037327 


1.000000 


0. 000000 


182 


2 


f 


-0. 147249 


-0. 143760 


0. 999023 


-0. 422065 


162 


2 


n 


-0. 195747 


-0. 186411 


0. 997777 


-0. 592810 


182 


2 


h 


0. 188673 


0. 190930 


0. 999964 


-1.425058 


182 


2 


J 


-0. 009634 


0. 009050 


0. 994672 


-0. 968134 


182 


2 


1 


0. 157129 


0. 158432 


0. 999989 


-1.458326 


182 


2 


m 


-0. 198272 


-0. 175150 


0. 994150 


-1. 148751 


182 


2 


r« 


-0. 190478 


-0. 159352 


0. 992020 


-1. 320885 



P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 « Cosine. #2 « Dice 
TW: Tern Weighting Schemes: See Result Page R-1 

correlation Coefficients: rju is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr is between 
• the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 « most relevant, 
4 « most non'relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Lev*!: A positive Z indicates that the second correlation i* higher 
than the first correlation {ry^ rju). If this Z is statistically 
significant as indicated by tfie asterisks, then resolving anaphors improves 
the system's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



NOMINAL SUBSTITUTES 



Correlation Coefficients 



Significance Level 



TW 



jr 



ur 



P > .05 



ISA 




a 


13£334 


-0. 132581 


0. 999986 


0. 160657 


164 




d 


0« 129099 


0. 143653 


0* 993570 


—ft 579173 


184 






0* 064221 


0. 095041 


0« 966992 


—0- 93183^ 


J 84 




h 


-0. 045491 


-0. 042586 


0« 999765 




1 an 




J 








~1. 009440 


164 




m 


0. 060914 


0. 102501 


0. 993606 


-0. 857047 


184 




n 


-0. 007330 


0. 023365 


0. 992337 


-1. 109195 


164 




a 


-0. 148937 


-0. 149158 


0. 999387 


0. 195327 


184 




b 


-0. 139155 


-0. 139231 


0. 999998 


0. 166794 


184 




c 


-0. 14SB7£ 


-0. 146048 


0. 999990 


0. 173707 


184 




d 


0. 1SB770 


0. 129955 


0. 986155 


-0. 032134 


184 


a 




0. 142193 


0. 157319 


0. 994299 


-0. 640365 


184 




f 


0. 039&£5 


0. 041512 


0. 966603 


-0. 051560 


164 


2 


9 


0. 121064 


0. 121775 


0. 985399 


-0.018749 


184 


£ 


h 


-0. 032087 


-0. 031778 


0. 999967 


-0. 169428 


164 




J 


0. 038815 


0. 050309 


0. 994543 


-0. 492561 


184 


a 


I 


-0. 004347 


-0. 004208 


1. 000000 


-1. 138199 


184 




m 


0. 078227 


0. 083962 


0. 990307 


-0. 249245 


184 




Vi 


-0. 024803 


-0. 005680 


0. 995324 


-0. 884545 



NOTES; 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Keasure: #1 « Cosine. 12 « Dice 
TW: Tern Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rj^ Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 > most relevant, 
4 * most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z indicates that the second correlation is higher 
than the first correlation {tA^y rjy). If this Z is statistically 
significant as Indicated by tne asterisks, then resolving anaphors improves 
the system's predications of relevance. 
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Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



PRO»YERBS 



Correlation Coefficients Significance Level 



Q 


S 




J** 


r 

ur 


z 


181 


1 


m 


-0.424357 


—0. 422512 


V. 999964 


« £k A ^^^^ 

—1. 198307 


lei 


1 


d 


-0. 745425 


-0. 751612 


0.999041 


1.047436 


lei 


1 


• 


-0.338200 


-0. 337852 


0.999810 


-0.096571 


lei 


1 


h 


-0. 084690 


-0.079805 


0.999592 


-0. 874708 


101 


1 


J 


-0.605974 


-0. 561823 


0.988776 


-1.741399 


lei 


1 


m 


-0.742554 


-0.746668 


0.998958 


0. 579180 


101 


1 


n 


-0. 692001 


-0. 698717 


0. 998276 


O. 797335 


lei 


2 




-0. 381796 


-0. 379436 


0.999941 


-1. 188190 


101 


a 


b 


-0. 328840 


-0. 327643 


0.999988 


-1.289199 


lei 


a 


c 


-0.381263 


-0. 379214 


0.999955 


-1. 185281 


101 


2 


d 


-0.701048 


-0.700325 


0.999297 


-0. 137691 


101 


2 


• 


-0. 339941 


-0. 339150 


0.999428 


-0. 126865 


101 


2 


f 


-0. 662621 


-0.662366 


0. 998983 


-0.038515 


101 


2 


e 


-0.701503 


-0. 700533 


0.999215 


-0. 175042 


101 


2 


h 


-0. 031633 


-0. 031677 


0.999999 


0. 144143 


101 


2 


J 


-0. 712239 


-0. 707026 


0. 998583 


-0. 698802 


101 


2 


1 


-0.001532 


-0.001532 


1.000000 


0.000000 


101 


2 


IN 


-0.731639 


-0. 729276 


0. 998978 


-0. 388272 


101 


2 


n 


-0.741866 


-0.741931 


0. 998306 


0. 008552 



P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Neasure: II « Cosine. 12 « Dice 
TW: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the uicr's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 • most relevant, 
4 • most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation irir> rjy). if this Z Is sutlstlcally 
significant as Indicated by the asterisks, then resolving anaphors Improves 
tho «yst«i*f prtdlcitlom of relevance. I93 
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A Stitlstlcal Conptrlson of the Relationship Between 
Unresolved Anaphors and User's Relevance JudgiRents with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



PRO-^BS 

Correlation Coefficients Significance Level 



Q 


s 


TH r. 


- ■ r^ 


r 


7 












ur 




103 


1 




0. 146264 


0. 144976 


0.999953 


0.665008 


103 


1 


d 


-0.05399a 


-0.053675 


0.999990 


-0. 340813 


103 


1 


• 


-0.000586 


-0.000586 


1.000000 


A ^^^^^^^^ 


103 


1 


h 


-0. 319705 


-0. 319717 




0.549665 


103 


1 


J 


-0.316556 


-0.385896 


0.997031 


0. 565491 


103 


1 


M 


-0.333513 


-0. 331906 


0.999771 


-0.366994 


103 


1 


n 


-0. 3a7ia9 


-0.325747 


0. 999783 


-0. 304195 


103 


s 




0.aS7744 


0.856450 


0.999941 


0. 608435 


103 


a 


b 


0. a95a30 


0.894669 


0.999966 


0.563865 


103 


a 


c 


0.aS6569 


0. 855414 


0.999951 


0. 598566 


103 




d 


-0. 003786 


-0. 004584 


0.999966 


0. 739361 


103 


2 


• 


-0.000586 


-0.000586 




0.000000 


103 


s 


f 


0.038656 


0. 038471 


0.999997 


0.739374 


103 


a 


B 


-0.0063S6 


009076 


0.999966 


0. 716064 


103 


a 


h 


-0. 314465 


-0. 314490 


la 000000 


1. 183470 


103 


a 


J 


-0.350606 


-0. 356106 


0. 996673 


0. 537535 


103 


a 


1 


-0.849676 


-0.249676 


1.000000 


0.000000 


103 


a 


M 


-0. 813615 


-0. 215067 


0. 999665 


0. 361936 


103 


a 


n 


-0. 848461 


-0. 844066 


0. 999797 


0. 402552 



P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: SiMllarlty Measure: #1 - Cosine. #2 * Dice 
TW: Tern Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaplwrs. 

Because the user's Judgments were scaled from low to high (1 ■ most relevant, 
4 « most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance Judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rjr> cju). If this Z Is sUtlstlcally 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the lystam's predications of relevance. j[94 
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A Statistical Coiiparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 



PRO»YERBS 



Co rr el a 1 1 on Coefficients 



Significance Level 



TU 



104 


1 


a 


104 


1 


d 


104 


1 


• 


104 


1 


h 


104 


1 


J 


104 


1 


M 


104 


1 


n 


104 


2 


a 


104 


& 


b 


104 


2 


c 


104 


2 


d 


104 


£ 


• 


104 


2 


f 


104 


2 


B 


104 


2 


h 


104 


2 


J 


104 


2 


1 


104 


2 


m 


104 


2 


n 



I. 071262 
1.623340 



-0.347650 

-O. 470712 
-0.602159 
-O. 415472 
-O. 153961 
-O. 161960 
-O. 149594 



-O. 069419 
-O. 624274 
-0.430760 
-0.347644 



-0.426036 
-O. 577165 
-0.595353 
-O. 176700 
-O. 472299 
-O. 017455 
-O. 565612 
-0. 364919 



ur 

0.999666 
0.999945 
0.999996 
O. 999996 

-0.470223 0.999250 
0.999642 
0.994046 
0.999640 
0.999965 
0.999666 
0.999966 
O. 999991 
0.999992 
0.999972 
0.999995 

O. 999610 
l.OOOOOO 
O. 999955 
O. 995359 



p > .05 



-O. 152407 
-O. 161479 
-O. 146053 
-0.593053 
-O, 427124 
577729 
-O. 596326 
-O. 179395 

-O. 475769 
-0. 017564 
-O. 565614 
-0. 356337 



-0.550944 

0. 507177 

1. 324326 
0.324667 

-0.063666 
O. 042653 
-1. 145406 
-O. 397725 
-0. 272674 
-O. 426003 
0. 722363 
li.241916 
0.761764 
O. 713660 
1.020076 
0. 691137 
1.230416 
0.115405 
-1.305987 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 « Cosine. #2 « Dice 
TU: Tern Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance judgment and the 
system's predicted romance based on unresolved anaphors. rjr Is between 
the user's relevance Judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 « most relevant, 
4 " most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation is higher 
than the first correlation (rjr> rju). If this Z Is sUtlstlcally 
significant as Indicated by the asterisks, then resolving anaphors Improves 
the system's predications of relevance. | qr 
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A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 



PRO-VERBS 



Correlation Coefficients Significance Level 



Q 


s 


T« r. 


r . 


r 


z 










jr 


ur 






1 


a 


-0.869170 


-0.269170 




0.000000 


107 


1 


d 


-0. 361714 


-0.362459 


9m 999991 


0. 775989 


197 


1 


• 


-0.001095 


-0.001095 






4»7 


1 


h 


0.152533 


0.152609 


0I 999996 


-0. 657191 


107 


1 


J 


*~0« 165056 


-0^166661 


0.999945 


0.652363 


107 


1 


M 


-0.327307 


-0.329623 


0.999947 


1.052114 


107 


1 


n 


-0. 263932 


-0.267654 


0.999670 


0.985711 


107 




a 


-0.290729 


-0.290729 




0.000000 


107 


a 


b 


-0. 265007 


-0.265007 




0.000000 


107 


& 


c 


-0.292260 


-0.292260 




0.000000 


107 




d 


-0. 356617 


-0.357469 


0.999969 


0. 803241 


107 


2 


• 


-0.001095 


091095 


1.000000 


0.000000 


107 


2 


f 


-0.357461 


-0.357901 


0.999997 


0.789957 


107 


2 


B 


-0.356079 


3S6067 


0.999991 


0. 801685 


107 


2 


h 


0.182394 


0. 161926 


0.999966 


0.401789 


107 


2 


J 


-0. 069937 


-0.092925 


0. 999B66 


0. 760167 


107 


2 


1 


0. 074756 


0.074394 


0. 999998 


0. 696253 


107 


2 


m 


-0. 315776 


-0. 317796 


0.999955 


0. 916822 


107 


2 


n 


-0.296199 


-0.299332 


0. 999907 


0. 985568 



P > .05 



NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200>299 on PsychlNFO 
S: SiMllarlty Measure: #1 « Cosine. §Z « Dice 
TW: Tern Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance Judgment and the 
systen's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance Judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's Judgments were scaled from low to high (1 « most relevant, 
4 • most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance Judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rwy rj^). If this Z Is sUtlstlcally 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the system's predications of relevance.^ 196 
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A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgaents: for Anaphoric Class 



PRO-VERBS 



Correlation Coefficients 



Significance level 



Til 



109 


1 


a 


109 


1 


d 


109 


1 


• 


109 


1 


h 


109 


1 


J 


109 


1 


M 


109 


1 


n 


109 


.a 




109 


z 


b 


109 


s 


e 


109 


2 


d 


109 


2 




109 


s 


f 


109 


2 


8 


109 


2 


h 


109 


2 


J 


109 


2 


1 


109 


2 


m 


109 


2 


n 



». 220S19 
I. 361494 



ur 



P > .05 



066046 -0.067262 



,131462 
328732 



.226134 
,233S10 
, 234593 
,285177 



-0.115245 
-0. 31 1233 
-0. 330916 
-0. 228134 
-0. 233510 
-0.234593 
-0.285957 



.293837 
.289566 



». 125951 
». 069214 
1.296651 
». 330370 



-0.294256 
-0.290261 
-0.069368 

-0.116716 
-0. 070142 
-0. 281933 
-0. 304518 




0.999744 
0. 998916 
0.999048 
0. 997145 



1.173908 

0.000000 

0.440498 
-1.775086 
-1.899918 
-i: 878653 

0.000000 
0.000000 

1.320403 
A flyitflMMyft 

1.295906 
1. 317960 
O. 319351 

-2.204416 <«««« 
O. 107587 
-1.868668 
-1.906291 



NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Slirilarlty Measure: II « Cosine. 12 « Dice 
TW: Tern Weighting Schenes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance Judgnent and the systcn's predicted relevance based 
on resolved anaphors. 

Because the user's Judgnents were scaled fron low to high (1 « nost relevant, 
4 « nost non-relevant) a strong negative correlation shows agreement 
between user's and systen's relevance Judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation {n^^ rj^). if this Z Is statistically 
significant as Indicated fey tne asterisks, then resolving anaphors Improves 
the syttMi's predications of relevance. ^ ^„ 
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P«9e 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgaents: for Anaphoric Class 

PRO'VERBS 



Correl atlon Coefficients 



Significance Level 



0 




TH r. 






2 










jr 






135 


1 


a 






8.999613 


1.784237 


135 


1 


d 


-•.797236 


-8.796217 


8.999967 


8.792867 


135 


1 


• 


-•.M1234 


-•.••1234 




8. 888888 


135 


1 


h 


-•.•66583 


•:•. •35626 _ 


<^ 998988 


8.885472 


135 


1 


J 


-0.633972 


-•. 6442^8 


8.968286 


8.276535 


135 


1 


M 


-•.7S6S76 


-8.B88^37 


8.999717 


8.878626 


135 


1 


r> 


-•.842132 


-•.811856 


8.995354 


-1.830233 


135 


a 


a 


-•. 637638 


-8.648359 


8.999258 


1.398371 


135 


2. 


b 


-8. 647^93 


-8.652388 


8.999766 


1.247222 


135 


s 


c 


-8.635248 


-8.644584 


8.999466 


1.416114 


135 


s 


d 


-8. 818316 


-8.819898 


8.999769 


0.257540 


135 


2 


mt 


-8. ••1234 


-8. 0^1234 


l.«W^O^ 




135 


s 


f 


-•. 618968 


-•.816917 


8.999919 


-0. 024199 


135 


2 


B 


-8.816188 


-8.817158 


•.99982:4 




135 


2 


h 


-8. 881788 


-8. •e2l2^ 


•. 999999 


1.315987 


135 


2 


J 


-8.766718 


-8. 776185 


•.995625 


8. 772382 


135 


2 


1 


-8.8^1329 


-8.801392 


« ^^^^^^^^ 


8. 479385 


135 


2 


M 


-8. 813643 


-8. 616586 


8.999667 


8.762869 


135 


2 


r» 


-8.826668 


-8. 797566 


8.995694 


-1.776553 



P > .05 



NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Sliillarlty Measure: #1 - Cosine. #2 « Dice 
TH: Tem Weighting Schenes: See Result Page R-1 
Correlation Coefficients: 



r^u Is between the user's relevance judgnent and the 
^ance based on unresolved anaphors. Is between 
the user's relevance judgnent and the system's predicted relevance based 
on resolved amphors. 



rju 

systOM's predicted releva 



Because the user's Judgnents were scaled fron low to high (1 « MOSt relevant, 
4 * Most non-relevant) a strong negative correlation shows agreenent 
between user's and systen's relevance judgnents. 

Significance Level: A positive Z Indicates thit the second correlation Is higher 
than the first correlation (rjr> rju)* If this Z Is sUtlstlcally 
significant as Indicated by tnc asterisks, then resolving anaphors Inproves 
the lystM's prtdlcatlom of rtlevance. \!^% 
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Appendix E.SS 



F«ge 

A Statistical Co«par1so« of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Jud9«e.its: for Anaphoric Class 



PRO-YERBS 



Correlation Coefficients 



Slonlflcaace Level 



ur 



P > .05 





1 


• 


192139 


-9.199532 


9.999963 


-1.337336 


142 


1 


d 


2SS097 


-9.297921 


9.999695 


9.699355 


142 


1 




'^v* 990662 




1.999999 




142 


1 


h 


-9.277^0 


299714 


9.799641 


-9.461599 


142 


1 


J 


-9.319695 


^299542 


9.624939 


-9. 216997 


142 


1 


n 


>9. 394197 


-9.243396 


9.954947 


995335 


142 


1 


n 


-9.211922 


-9.112939 


9.936995 


-1.263212 


142 


2 


• 


-9. 324319 


-9.321492 


9.999945 


-1.265263 


142 


2 


b 


-9.435629 


-9.434697 


9.999992 


-1.299647 


142 


2 


c 


-9.355969 


-9.353569 


9.999961 


-1.251661 


142 


2 


d 


-9.339456 


-9.349976 


9.999664 


9.462161 


142 


2 


• 






1.999999 




142 


2 


f 


-9.452453 


-9.453993 


9.999974 


9.429176 


142 


2 


8 


-9.352275 


-9.353636 


9.999997 


9. 464213 


142 


2 


h 


-9.277419 


-9.927969 


9. 129421 


-9.653365 


142 


2 


J 


-9. 316247 


-9.335936 


9.799655 


9. 137673 


142 


2 


1 


-9.255325 


-9. 991435 


9.395597 


-1.936994 


142 


2 


n 


-9. 359397 


-9. 321676 


9. 959949 


-9.696619 


142 


2 


n 


-9.254617 


-9.194395 


9. 943519 


-9. 694963 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Slnflarlty Measure: #1 - Cosine. #2 » Dice 
TH: Teni Weighting SchcMs: See Result Page R-1 

Correlation Coefficients: ri„ Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance Judgneiit and the systcn's predicted relevance based 
on resolved anaphors. 

Because the user's judgaents were scaled fron low to high (1 « nost relevant, 
4 • aost non-relevant) a strong negative correlation shows agreonent 
between user's and systen's relevance Judgnents. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rjr> rju). if this 2 Is sutlstlcally 
significant as Indicated by the asterisks, then resolving anaphors Improves 
the lystM's predications of relevance. ^ 
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Page 

A Statistical Ceap<>ir1soii of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgaents: for Anaphoric Class 



PRO-IERBS 



Correlation Coefficients Significance Level 





s 


TM r. 


- • r - 


r 


z 








a« 


jr 


ur 




156 


1 




•.•3B3ia 


0.037274 


0.999B61 


0.234405 


ISB 


1 


d 


-B. 16291 1 


-0.163220 


0.9999BS 


0.217269 


ISA 


i 


• 


-•.M1932 


-0.001038 




0.000000 


ISB 


1 


h 


>«.2S1133 


-0.26B30B 


0.999643 


1.034222 


ISB 


1 


J 


-B. 172647 


-0. 18S305 


0.991290 


0.35BB75 


ISB 


1 


ii 


-B. 212637 


-0.219746 


0.999B2B 


1.451400 


ISB 


1 


n 


-B. 19B37B 


-0.209S90 


0.992926 


0.616191 


ISB 


2 


m 


•.B90669 


0.0B932B 


0.999B09 


0. 2S747B 


ISB 


2 


to 


B. 17447S 


0. 173910 


0.999972 


0.2B5S31 


ISB 


2 


c 


B. 11613B 


0. 11S103 


0.999B7S 


0. 244216 


ISB 


2 


d 


113967 


-0.114001 


0.999B6B 


O.0079B4 


ISB 


2 


m 


-0.001032 


-0.001032 






ISB 


2 


f 


B. 016314 


0.01S779 


0.999971 


0.262B13 


ISB 


2 


8 


•-0.OB93O6 


-0.0B9416 


0.999B9B 


0.02B95B 


ISB 


2 


h 


-B.136S51 


-0. 175037 


0.997409 


2.004B3S 


ISB 


2 


3 


-0. 103790 


-0.11BB37 


0. 99B761 


1.136096 


ISB 


2 


I 


-0.007262 


-0.020390 


0.999677 


1.932696 


ISB 


2 


m 


-0. 131206 


-0.137B06 


0.99964B 


1. 426312 


ISB 


2 


n 


-0. 12077B 


-0.135942 


0.996106 


0. 647920 



P > .05 



<««*« 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: SiMllarlty Measure: «1 « Cosine. #2 « Dice 
TU: Tem Weighting Schcnes: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgnent and the systcn's predicted relevance based 
on resolved amphors. 

Because the user's judgments were scaled froM low to high (1 « most relevant, 
4 • most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates thit the second correlation Is higher 
than the first correlation (rjr> rju). If this Z Is sUtlstlcally 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the fystM's predications of relevance. onn 
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Ptge 

A Statistical Coaparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgaents with Resolved 
Anaphors and User's Relevance Judgaents: for Anaphoric Class 



PRO-VERBS 



Correlation Coefficients 



Significance Level 



TM 



179 
170 
170 
170 

170 
170 
170 
170 
170 
170 
170 
170 
170 
170 
170 
170 
170 
170 
170 



1 
1 
1 
1 

1 
1 
1 

2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 



h 
J 



b >0 

c -0, 

d -0, 
• -0. 
f -0. 

g -0. 

h 0. 

J -a 

1 0. 
M -0. 

n -0. 



.716613 
.731091 
.001109 

.166612 

.594562 
.636259 
.515708 
.678365 
.637558 
.678050 
, 691708 
001109 
696939 
689511 
153761 
, 553947 
045678 
623943 
533728 



-0.716201 
-0.729330 

-0.001109 

0.165300 

-0.595376 
-0.635420 
-0.516671 
-0.678112 
-0.637414 
-0.677830 
-0.690993 

-0.001109 

-0.696486 
-0. 688913 

0. 153380 
-0. 553885 

0. 046002 
-0.623468 
-0. 534747 



ur 

0.999987 
0.999896 

1.000000 

0.999969 

0.999847 

0.999983 
0.999991 
0.999979 
0.999996 
0.999983 
0.999972 
1.000000 
0.999991 
0.999978 
0.999986 
0. 999891 
0.999997 
0.999987 
0.999992 



P > .05 



-0.744580 

0.738479 

0.252098 
-0.789984 

1. 138468 
-0. 249519 
-0.274505 
-0.262757 
-0.570539 

0.000000 

-0.628686 
-0.532944 

0. 312437 
-O. 022236 
-O. 622328 
-O. 515001 

1.261938 



MOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Sinllarlty Measure: II » Cosine. 12 « Dice 
TU: TenN Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 « most relevant, 
4 ■ most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level : A positive Z Indicates that the second correlation Is higher 
than the first correlation (rjr> rj^}. if this Z Is statistically 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the DfStM's prtdlcatlons of relevance. 
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Ptge 

A Statistical CoaparlsoR of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgaents with Resolved 
Anaphors and User's Relevance Judgments : for Anaphoric Class 



PRO»YERBS 



Correlation Coefficients Significance Level 



Q 


s 


T« r. 






z 








J" 




ur 






t 


a 


194413 


—V. 


O. 999984 


-1.386175 


182 


1 


d 


-•.221779 


-0.204120 


0. 997707 


-1.405551 


182 


1 


• 




-0.000462 






182 


1 


h 


a. 185728 


0.185760 




-0. 037143 


182 


1 


J 


-0.M8062 


0.012035 


0.996180 


-1.216836 


182 


1 


M 


-0. 187365 


-0.156366 


9.992670 


-1.371693 


182 


1 


n 


-0.168005 


-0. 131268 


0.990061 


-1.316344 


182 


2 


a 


-0. 174304 


-0. 172579 


0.999979 


-1.414123 


182 


2 


b 


-0. 101067 


-0.100631 


0.999999 


-1.424118 


182 


2 


c 


-0. 137425 


-0.136297 


0.999991 


-1.417226 


182 


2 


d 


-0.223362 


-0. 207851 


0. 998328 


-1.446237 


182 


2 


• 


-0. 037327 


-0.037327 


1.000000 


O.OOeOO0 


182 


2 




-0. 147249 


-0.138295 


0.999463 


-1.457952 


182 


2 


9 


-0. 195747 


-0. 181985 


0. 998714 


-1.456521 


182 


2 


h 


0. 188673 


0. 186046 


0. 999861 


0. 846242 


182 


2 


J 


-0.009834 


0.008408 


0. 995247 


-0. 990166 


182 


2 


1 


0. 157129 


0. 156170 


0. 999965 


0.615318 


182 


2 


IK 


-0. 198272 


-0. 172524 


0. 994889 


-1.367066 


182 


2 


n 


-0. 190478 


-0. 161261 


0. 992680 


-1.294802 



p > .OS 



NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: SiRllarlty Measure: #1 » Cosine. #2 » Dice 
TW: Tern Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rw between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 ■ most relevant, 
4 • most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation {rw> rju). If this Z Is sUtlstlcally 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the lystM's predications of relevance. pnp 
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Page 

A Statistical Coaparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgiaents: for Anaphoric Class 



PRO-VERBS 



Correlation Coefficients Significance Level 





S 


TW r. 




r 


<• 

z 












ur 




188 


1 




-0. 297S36 


-0.299409 


0. 999723 


0.416233 


lae 


1 


d 


-0. 475084 


-0.485908 


0. 999376 


1.677809 


189 


1 


• 


-0.4848S4 


-0. 484654 


1 • 000000 


0. 000000 


lae 


1 


h 


-0. 234297 


-:0. 250428 


0.998063 


1.327498 


180 


1 


J 


-0. 39S090 


•^.398504 


0. 998871 


0. 390883 


180 


1 


m 


-0. 473372 


-0.482114 


0. 999679 


1.868891 


180 


1 


n 


-0. 428501 


-0.432483 


0.999846 


1.235124 


180 


2 


a 


-0.219548 


-0.222590 


0.999568 


0.530523 


180 


2 


b 


-0. 157482 


-0. 159036 


0.999904 


0. 573887 


180 


2 


c. 


-0. 213261 


-0.215989 


0. 999684 


0. 554789 


180 


2 


d 


-0. 475058 


-0. 487901 


0. 999342 


1.914103 


180 


2 


• 


-0. 434G32 


-0.434632 


1.000000 


0. 000000 


180 


2 


f 


-0. 393963 


-0. 404198 


0.999606 


1.916450 


180 


2 


9 


->0. 468597 


-0.481172 


0. 999380 


1.925501 


180 


2 


h 


-0. 352302 


-0. 377404 


0. 991004 


0.997600 


180 


2 


J 


-0. 449142 


-0. 454957 


0. 999533 


1.051901 


180 


2 


1 


-0. 126480 


-0. 138984 


0.999041 


1.437095 


180 


2 


m 


-0. 476596 


-0. 486384 


0. 999647 


1.983032 


180 


2 


r» 


-0. 445123 


-0. 450480 


0. 999766 


1.353539 



<««*» 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: II • Cosine. 12 • Dice 
TW: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rjy Is between 
the user's relevance Judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's Judgments were scaled from low to high (1 » most relevant* 
4 * most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance Judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rjr> rjy). If this Z Is statistically 
significant at Indicated by tne asterisks, then resolving anaphors Improves 
the lyttiM't prtdlcatlons of rtltvance. 203 
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Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric CUss 

PRO-VERBS 



Correlation Coefficients 



Significance Level 



Q 


s 


TW r. 






z 








JU 




ur 




184 


1 




-0. 132394 


-0. 132581 


0. 999986 


0. 160857 


184 


1 


d 


8.129899 


0. 129805 


0. 999996 


-1.072488 


184 


1 


c 


0. 064221 


0. 064221 




0. 000000 


184 


1 


h 


-e. 845491 


-0. 045478 


0. 999956 


-0. 006535 


184 


1 


J 


0. 883093 


0.006652 


0. 999801 


-0. 797005 


184 


1 


m 


0. 080914 


0.082766 


0. 999986 


-1.588O80 


184 


1 


n 


-0. 007330 


-0.002700 


0. 999864 


-1.256942 


184 


2 


a 


-O. 148937 


-0. 149156 


0. 999987 


0. 19S327 


184 


& 


b 


-e. 139155 


-0. 139231 


0. 999998 


0. 166794 


184 


2 


c 


-0. 145872 


-0. 146048 


0. 999990 


0. 173707 


184 


2 


d 


0. 128770 


0. 129277 


0. 999996 


-0. 851096 


184 


2 


• 


0. 142193 


0. 142193 


1.000000 


0. 008000 


184 


2 


f 


0. 039825 


0.039735 


0. 999999 


-0. 414221 


184 


2 


D 


0. 121084 


0. 121514 


0. 999997 


-0. 832490 


184 


2 


h 


-0. 032087 


-0. 033874 


0. 999309 


0. 215014 


184 


2 


J 


0.038815 


0.039882 


0. 999887 


-0. 317027 


184 


2 


1 


-0. 004347 


-0. 004624 


0. 999980 


0. 195383 


184 


2 


m 


0. 078227 


0. 076962 


0. 999980 


-0. 516291 


184 


2 


n 


>0. 024803 


-0. 022960 


0. 999965 


-0. 983740 



p > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 - Cosine. #2 * Dice 
TW: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rj|> Is between 
the user's relevance Judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's Judgments were scaled from low to high (1 ■ most relevant, 
4 * most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance Judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rjr> rju). If this Z Is statistically 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the system's prtdlcatlons of relevance. 
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Appendix E>61 
Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



INDEFINITES 



Correlation Coefficients Significance Level 



0 


s 


TW r. 




1^ 


7 








JU 


jr 


%r 




101 


1 


a 


-0. 4243S7 


-0. 422512 


0. 993964 


—1. 298307 


101 


1 


d 


-0. 745425 


-0. 752322 


0. 997931 


0. 608034 


101 


1 


• 


'0. 338200 


-0. 341814 


0. 999590 


0. 682320 


101 


1 


h 


-0. 084690 


-0. 079382 


0. 999586 


-0. 943369 


101 


1 


J 


>0. 605974 


-0. 560459 


0. 986712 


-1.659860 


101 


1 


m 


-0. 742554 


-0.741852 


0. 996874 


-0. 067649 


101 


1 


ft 


-0. 692001 


-0. 683252 


0. 996031 


-0. 681516 


101 


2 




-0.381796 


-0. 379436 


0. 999941 


-:. 188190 


101 


& 


b 


-0. 328840 


-0. 327643 


0. 999988 


-1.289199 


101 


2 


c 


-0. 381283 


-0. 379214 


0. 999955 


-1. 185281 


101 


£ 


d 


-0. 701048 


-0. 695983 


0.997959 


-0. 560160 


101 


& 


• 


-0. 339941 


-0. 344668 


0. 998915 


0. 549529 


101 


2 


f 


-0. 662621 


-0. 656817 


0. 997499 


-0. 552888 


101 


S 


g 


-0.701503 


-0. 695683 


0. 997813 


-0. 621423 


101 


£ 


h 


-0.031633 


-0. 031647 


0. 999998 


0. 040856 


101 




J 


-0. 712239 


-0. 696749 


0.997213 


-1.405118 


101 


a 


1 


-0. 001532 


-0. 001532 


1.000000 


0. 000000 


101 




m 

• 


-0. 731639 


-0. 720449 


0. 997470 


-1. 121817 


101 


2 


n 


-0. 741866 


-0. 722653 


0. 996692 


-1.620531 



P > .05 



NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 * Cosine. #2 « Dice 
TM: Term Weighting Schemes: See Result Page R-1 

Correlation CoefflclcnU: rju Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 » most relevant, 
4 • most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z indicates that the second correlation Is higher 
than the first correlation ir^^ rju). If this Z Is statistically 
significant as Indicated ty the asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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Appendix E-62 
Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



INDEFINITES 



Correlation Coefficients Significance Level 



n 

V 


e 

d 






ur 


7 


103 


1 




0. 146264 


0. 144976 


0. 999953 


0. 665002 


103 


1 


d 


-0. 053992 


-0. 066044 


0. 999426 


1.743906 


103 


1 


• 


-0. 000526 


-0. 000526 


1.000000 


0. 000000 


103 


1 


h 


-0. 319705 


-0. 319743 


1.000000 


1.210674 


103 


1 


J 


-0. 316556 


>0. 329609 


0.996632 


0. 644967 


103 


1 


m 


'0. 333513 


-0.344174 


0. 999021 


1.241036 


103 


1 


r» 


-0. 327129 


-0. 336017 


0. 999064 


1.056705 


103 


a 


• 


0. 257744 


0. 256450 


0. 999941 


0. 60243S 


103 


& 


b 


0. 295230 


0. 294669 


0. 999966 


0. 563265 


103 


2 


c 


0. 256569 


0. 255414 


0. 999951 


0. 592566 


103 


& 


d 


-0. 003726 


-0.010393 


0. 999642 


1.637296 


103 


s 


• 


-0. 000526 


-0. 000526 


1.000000 


0. 000000 


103 


& 


f 


0. 032656 


0. 029446 


0. 999957 


1.601669 


103 


2 


9 


>0. 006356 


-0.014515 


0. 999666 


1.644286 


103 


2 


h 


-0. 314465 


-0. 314500 


1.000000 


1.429243 


103 


2 


J 


>0. 350606 


-0. 356694 


0. 996611 


0. 780160 


103 


2 


1 


-0. 249676 


-0. 249676 


1.000000 


0. 000000 


103 


2 


m 

• 


-0. 213615 


-0.221119 


0. 999677 


1.433116 


103 


2 




-0. 242461 


-0. 249939 


0. 999593 


1.316620 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 « Cosine. #2 « Dice 
TW: Tern Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on rtsolvcd anaphors. 

Because the user's Judgments were scaled from low to high (1 • most relevant, 
4 > most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance Judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (r4r> rju)* If this Z Is statistically 
significant as Indicated by tnc asterisks* then resolving anaphors Improves 
the liystcm's predications of relevance. 
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Appendix E-63 
Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



INDEFINITES 



Correlation Coefficients 



Significance Level 



Q 


< 


204 




10A 




104 




104 




104 




104 




104 




104 




104 


£ 


104 




104 




104 


2 


104 


a 


104 


2 


104 


2 


104 


2 


104 


2 


104 


2 


104 


2 



TW 



a -0 

d -0 

• -0. 

h -0. 

J -0. 

m -0. 
n -0. 

» -0. 

b -0. 

C -0. 

d -0. 

• -0. 

f -0. 

B -0» 
h -0. 

J -0. 
1 -0. 

M -0. 

ri -0. 



. 0712£2 
. £23340 
429965 
347650 

470712 
602159 
415472 
153961 
161980 
149594 
592005 
426036 
577165 
595353 
178700 
472299 
017455 
565612 
384919 



ur 



p > .05 



-0. 071204 
-0. 627242 
-0. 432473 
-0. 347405 
-0. 468763 
-9. 602996 
-0. 384340 
-0. 154278 
-0. 162270 
-0. 149793 
-0. 594546 
-0. 428956 
-0. 578412 
-0. 597809 
-0. 179124 
-0. 476056 
-0.017455 
-0. 566576 
-0. 357451 



0. 999904 


-0.018778 


0. 999709 


0. 905140 


0. 999958 


1. 318606 


0. 999997 


-0. 486152 


0. 999504 


-0. 313059 


0. 999857 


0. 276386 


0. 994260 


-1.385539 


0. 999859 


0. 080141 


0. 999969 


0. 165467 


0. 999884 


0. 059193 


0. 999882 


0. 899312 


0. 999934 


1.235247 


0. 999972 


0. 888252 


0. 999891 


0. 908231 


0. 999996 


0. 679858 


0. 999869 


1. 151360 


1.000000 


0. 000000 


0. 999931 


0. 443463 


0. 995310 


-1.341035 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 « Cosine. 12 « Dice 
TU: Tern Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rjy is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance Judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's Judgments were scaled from low to high (1 « most relevant, 
4 ■ most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance Judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (r4r> rj^). If this Z Is sUtlstlcally 
significant as Indicated by the asterisks, then resolving anaphors Improves 
the iiystcm's predications of relevance. 
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Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



INDEFINITES 



Correlation Coefficients 



Significance Level 



TW 



ju 



ur 



p > .05 



107 




a 


-0.269170 


-0.269170 


1.000000 


0. 000000 


107 




d 


-0. 361714 


-0. 362459 


0. 999991 


0. 775989 


107 




• 


-0. 001095 


-0. 001095 


1.000000 


0. 000000 


107 




h 


0. 152533 


0. 152809 


0. 999996 


-0. 657191 


107 




J 


-0. 165056 


-0. 186661 


0. 999945 


0. 652383 


107 




m 


-0. 327307 


-0. 329623 


0. 999947 


1.052114 


107 




n 


-0. 263932 


-0. 267654 


0. 999670 


0.985711 


107 




• 


-0. 290729 


-0. 290729 


1.000000 


0. 000000 


107 


£ 


b 


-0. 265007 


-0. 285007 


1.000000 


0. 000000 


107 


2 


c 


-0. 292260 


-0. 292260 


1.000000 


0. 000000 


107 


2 


d 


-0. 356617 


-0. 357489 


0. 999989 


0. 803241 


107 


S 


• 


-0. 001095 


-0. 001095 


1.000000 


0. 000000 


107 


S 


f 


-0. 357461 


-0. 357901 


0. 999997 


0. 769957 


107 


2 


9 


-0. 356079 


-0. 356667 


0. 999991 


0. 801665 


107 


2 


h 


0. 182394 


0. 181926 


0. 999988 


0. 401769 


107 


2 


J 


-0. 069937 


-0. 092925 


0. 999868 


0. 760167 


107 


2 


1 


0. 074756 


0. 074394 


0. 999998 


0. 696253 


107 


2 


. m 


-0. 315776 


-0. 317798 


0. 999955 


0. 916622 


107 


2 


n 


-0. 296199 


-0. 299332 


0. 999907 


0. 965568 



NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 « Cosine. 12 « Dice 
TH: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr 1$ between 
the user's relevance Judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 « most relevant, 
4 « most non-relevant) a strono negative correlation shows agreement 
between user's and system's rtievance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (r{r> rju). If this Z Is statistically 
significant as Indicated ty tne asterisks, then resolving anaphors Improves 
the lyrstem's predications of relevance. 
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Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



INDEFINITES 



Correlation Coefficients Significance Level 



A 

w 


r 
O 






•"ur 


7 


P > 


103 




a 


-0. 220S19 


-0.219931 


0. 999868 


-0. 199317 




1(99 


1 


d 


-0. 360544 


-0. 360360 


0. 999729 


-0. 045736 








• 


-0. 000545 


-0. 000545 


1.000000 


0. 000000 








h 


-0. 0G604& 


-0. 066351 


0. 999942 


0. 152409 




;09 




• J 


-0. 131462 


-0. 122050 


0. 999671 


-1.984724 


<#»»« 


109 




m 


-0. 328732 


-0. 323165 


0. 997027 


-0.410927 




109 






-0. 358384 


-0. 342772 


0. 995164 


-0. 908314 




109 




« 


-0. 228134 


-Z. 227383 


0. 999846 


-0. 237046 




109 


£ 


b 


-0. 233510 


-0. 233074 


0. 999920 


-0. 190643 




109 




c 


-0. £34593 


-0. 233846 


0. 999866 


-0. 253052 




109 


Z 


d 


-0. 285177 


-0. 286415 


0. 999469 


0. 213575 




109 


a 


• 


-0. 000545 


-0. 000545 


1.000000 


0. 000000 




109 




f 


-0. 293837 


-0. Z9^S^7 


0. 999285 


0. 105794 




109 




9 


-0. 289566 


-0. 290706 


0. 999402 


0. 185584 




109 


£ 


h 


-0. 068850 


-0. 068648 


0. 999993 


-0. 284085 




109 


a 


J 


-0. 125951 


-0. 120357 


0. 999848 


-1.736169 




109 




1 


-0. 069214 


-0. 069423 


0. 999747 


0. 050043 




109 


z 


.m 


-0. 296651 


-0. 281066 


0. 998933 


-1.869049 




109 


2 


n 


-0. 330370 


-0. 301879 


0. 996543 


-1.908738 





NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: 11 « Cosine. 12 • Dice 
TH: Term Weighting Schemes: See Result Page R*l 

Correlation Coefficients: rju Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 « most relevant, 
4 « most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (r4|.> r^^). If this Z Is sUtlstlcally 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the vstcm's predications of relevance. 

209 



Page 203 



Appendix E-66 
Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



INDEFINITES 



Correlation Coefficients 



Significance Level 



TW 



135 

135 

135 

135 

135 

135 

135 

135 

135 

135 

135 

135 

135 

135 

135 

135 

135 

135 

135 



1 
1 
1 
1 
1 
1 
1 

a 
a 

2 
S 

a 
a 
z 
s 

2 



h 

J 
m 
n 
a 

b 
c 
d 

f 

5 
h 

J 
1 
m 



-e. 590S94 
-0. 797236 
-0. 091234 
-0. 026503 
-0. 633972 
-0. 796876 
-0. 842132 
-0. 637630 
-0. 647093 
-0. 635240 
-0.818316 
-0. 001234 
-0. 818960 
-0.816168 
-0. 001768 
-0. 766710 
-0.001329 
-0.813643 
-0. 826860 



-0. 600904 
-0. 772840 
-0.001234 
-0. 035609 
-0. 637440 
-0. 778377 
-0. 795701 
-0. 648359 
-0. 652380 
-0. 644584 
-0. 809892 
-0.001234 
-0. 817350 
-0. 807974 
-0. 002120 
-0. 761965 
-0.001392 
-0. 807226 
-0. 797037 





z 


ur 




0. 999(»13 


1.704237 


0. 991218 


-1. 140052 


1.000000 


0. 000000 


0. 998909 


0. 803948 


0.980311 


0. 093623 


0. 996115 


-1.275569 


0. 992775 


-2. 013543 


0. 999258 


1.390371 


0. 999766 


1.247222 


0. 999466 


1.418114 


0. 995072 


-0. 591501 


1.000000 


0. 000000 


0. 998138 


-0. 188927 


0. 996205 


-0. 650900 


0. 999999 


1. 315987 


0. 993959 


-0. 275507 


1.000000 


0. 479305 


0. 996466 


-0. 530789 


0. 995023 


-1.717193 



P > .05 



<«•*• 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNPO 
S: SiMflarlty Measure: #1 - Cosine. 12 - Dice 
TH: Teni Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rj^ Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rj^ Is between 
the user's relevance judgment and the system's predicted relevance based 
on rtsolved anaphors. 

Because the user's judgments were scaled from low to high (1 • most relevant, 
4 ■ most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Signlflcanct Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rjr> r^u). If this Z Is sUtlstlcally 
significant as Indicated by tne asterisks* then resolving anaphors Improves 
the fyrstem's predications of relevance. 
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Page 

Comparison of the Relationship Between 

and User's Relevance Judgments with Resolved 

Relevance Judgments: for Anaphoric Class 



INDEFINITES 

Correlation Coefficients Significance Level 



TW 



ir 



ur 



P > .05 



142 




a 


-0. 192139 


142 




d 


-0. 255597 


142 




• 


-0. 000662 


142 




h 


-0. 277455 


142 




J 


-0. 318605 


142 




M 


-0.304107 


142 




n 


-0.211922 


142 


2 


a 


-0. 324310 


142 


2 


b 


-0. 435620 


142 


2 


c 


-0. 355960 


142 


2 


d 


-0. 339456 


142 


2 


• 


-0. 000662 


142 


2 


f 


-0. 452453 


142 


2 


S 


-0. 352275 


142 


2 




-0. 277419 


142 


2 


J 


-0. 316247 


142 


2 




-0. 255325 


142 


2 


ff. 


-0. 359307 


142 


2 


• ri 


-0.254817 



-0. 189532 
-0. 269432 
-0. 000662 
-0. 209714 
-0.291969 
-0. 254739 
-0. 120100 
-0. 321402 
-0. 434607 
-0. 353568 
-0. 346993 
-0. 000662 
-0. 455626 
-0. 359016 
-0. 027069 
-0. 338219 
-0.001435 
-0. 327471 
-0. 198757 



0. 999963 

0. 999431 

1. 000000 
0. 760641 
0. 824910 
0. 955165 
0. 940354 
0. 999945 
0. 999992 
0. 999961 
0. 999751 
1.000000 
0. 999944 
0. 999799 
0. 120421 
0. 799887 
0. 395597 
0. 959547 
0. 944209 



-1.337336 
1. 819247 

0. 000000 

-0. 461590 
-0. 206963 
-0. 747293 
-1. 175800 
-1.265263 
-1.209647 
-1.251661 
1.532558 

0. 000000 

1.419687 
1.531634 
-0. 853365 
0. 161175 
-1.038994 
-0. 519066 
-0. 750871 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: II « Cosine. 12 « Dice 
TW: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rj^ Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance Judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's Judgments were scaled from low to high (1 * most relevant, 
4 « most non-relevant) « strong negative correlation shows agreenent 
between user's and system's relevance Judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation {rwy rju)* If this Z Is statistically 
significant as Indicated ty tne asterisks, then resolving anaphors Improves 
the lystM's predications of relevance. ^ ^ ^ 
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Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



INDEFINITES 



Correlation Coefficients Significance Level 



Q 


S 


TW 






'"ur 


2 P > 


158 


1 


a 


0. 038318 


0. 037274 


0« 9936G1 




158 


I 


d 


-0. 162911 


-0. 168945 


0. 999338 




158 


I 


• 


-0. 001032 


-0. 001032 


1.000000 


0. 000000 


158 


2 


h 


-0. 281 133 


-0. 268266 


0. 999657 


1.047634 


158 


1 


J 


-0. 172847 


-0. 188528 


0. 991 179 


0. 448889 


158 


1 


m 


-0.212637 


-0. 224449 


0. 999513 


1.436359 


158 


1 


n 


-0. 190370 


-0. 210647 


0. 992773 


0. 649489 


158 


& 


a 


0. 090669 


0. 069326 


0. 999609 


0. 257478 


158 


a 


3 


0. 17A475 


0. 173910 


0. 999972 


0. 285531 


158 




c 


0. 116130 


0. 115103 


0. 999675 


0. 244216 


158 


2 


d 


-0. 113967 


-0. 127640 


0. 996101 


0.591873 


158 


s 


• 


-0.001032 


-0. 001032 


1.000000 


0. 000000 


153 


s 


f 


0.016314 


0. 000194 


0. 996324 


0. 703471 


156 


2 


S 


-0. 089306 


-0. 104073 


0. 995964 


0.617744 


158 


s 


h 


-0. 136551 


-0. 174165 


0. 997516 


2.002268 <«*«« 


158 


2 


J 


-0. 103790 


-0. 126435 


0. 997999 


1.345056 


158 


2 




-0. 007262 


-0. 020129 


0. 999681 


1.904769 


158 


2 




-0. 131206 


-0. 148421 


0. 997938 


1.010909 


158 


2 


• 


-0. 120778 


-0. 138366 


0. 995699 


0. 732335 



.05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Neasure: II - Cosine. #2 » Dice 
TU: Term Weighting Schemesi See Result Page R-1 

Correlation Coefficients: r«^^ Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rjf Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 • most relevant, 
4 ■ most non-relevant) « strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rw^ rju). If this Z Is statistically 
significant as Indicated by tne asterisks, then resolving amphors Improves 
the lystM's predications of relevance. 
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Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



INDEFINITES 



Correlation Coefficients 



Significance Level 



TW 



170 1 a -0.716613 

:.7«i 1 d -0.731051 

170 : • -0.001109 

170 1 h 0.166612 

170 1 J -0.594562 

170 1 m -0.636259 

170 I n -0.515708 

170 2 a -0.678365 

170 2 5 -0.637558 

170 2 e -0.678090 

170 2 d -0.691708 

170 2 • -0.001109 

170 2 f -0.696939 

170 2 g -0.689511 

170 2 n 0.153761 

170 2 J -0.553947 

170 2 i 0.045678 

:70 2 m -0.623943 

170 2 *ri -0.533728 



-0. 716201 
-0. 729927 
-0.001109 

0. 165325 
-0. 600554 
-0. 636255 
-0. 516341 
-0.678112 
-0. 637414 
-0. 677830 
-0. 691445 
-0.001109 
-0. 696701 
-0. 689362 

0. 153339 
-0. 555406 

0. 045931 
-0. 624053 
-0. 535895 



"ur 


Z 


W 1 




•Cm 999987 


-0. 500158 


A QQQfiCO 

w. Unabo 


-0. 437500 


1.000000 


0. 000000 


0. 999969 


0.718610 


0. 998838 


0. 667406 


0. 999905 


-0. 001826 


0. 999802 


0. 790715 


0. 999979 


-0.249519 


0. 999996 


-0. 274505 


0. 999983 


-0. 262757 


0. 999960 


-0. 177914 


1.000000 


0. 000000 


0. 999988 


-0. 288599 


0. 999967 


-0. 1 10846 


0. 999985 


0. 343563 


0. 999790 


0. 372142 


0. 999997 


-0. 476422 


0. 999959 


0. 067787 


0. 999932 


0. 941700 



P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: II « Cosine. #2 « Dice 
TU: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rw Is between 
the user's relevance Judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's Judgments were scaled from low to high (1 - most relevant, 
4 « most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance Judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (r4p> rju). If this Z Is statistically 
significant as Indicated by the asterisks, then resolving anaphors Improves 
the system's pred* atlons of relevance. ^ . . 
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Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



INDEFINITES 



Correlation Coefficients Significance Level 



0 
y 


s 






r 

'^ur 


z 


D > 


180 




* 


-0. 237536 


-0. 299409 


0. 999723 


0.416233 




aQV 


I 


a 


-0. 475084 


—0. 48SS0S 


0. 996877 


0. 746074 




180 




• 


-0. 484654 


-0. 483529 


0. 999452 


-0. 194078 




180 


•* 


h 


-0. 234297 


-0. 250120 


0. 998018 


1.287587 




;80 


1 


J 


-0. 395090 


-0. 397840 


0. 997360 


0. 206057 




180 






-0. 473372 


-0. 482802 


0. 996953 


0. 683441 




180 


1 


r> 


-0. 428501 


-0. 436343 


0. 997985 


0. 681462 




180 


2 




-0.219548 


-0. 222590 


0. 999566 


0. 530523 




:ia0 


Z 


b 


-0. 157462 


-0. 159036 


0. 999904 


0. 573687 




.180 


■ 2 


c 


-0.213261 


-0.215989 


0. 999684 


0. 554769 




2 80 


2 


d 


-0. 475056 


-0. 497852 


0. 998381 


2. 139282 


<**»« 


180 


2 


• 


-0. 434632 


-0.431416 


0. 998580 


-0. 334327 




180 


2 




-0. 393963 


-0.417118 


0. 998748 


2. 385579 


<***» 


180 




C 


-0. 468597 


-0. 492359 


0. 998440 


2. 249597 


<«»»« 


180 


2 


h 


-0. 352302 


-0. 377037 


0. 990957 


0. 980613 




180 


2 




-0. 449142 


-0. 459918 


0. 999212 


1.482148 




130 


2 


1 


-0. 126480 


-0. 138760 


0. 999033 


1.405332 




;a0 


2 


m 


-0. 476596 


-0. 494443 


0. 998821 


1. 982610 


<«**« 


180 


2 


•n 


-0.445123 


-0. 457971 


0. 998923 


1.508164 





NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: II « Cosine. #2 « Dice 
TH: Term Weighting Schemes: See Result Page R-1 

Correlation Cbefflclents: rju Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 • most relevant, 
4 « most non-relevant) i strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (r4|.> rju). If this Z Is statistically 
significant as Indicated by tne asterisks* then resolving anaphors Improves 
the system's predications of relevance. 
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Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



INDEFINITES 



Correlation Coefficients 



Significance Level 



Q 


S 


TW r. 
















ur 


1 A:^ 
IOC 








—0. 19<^9jk6 


0. 999984 


•ft 


« 


d 


-0. 221779 




0. 999972 


18£ 




• 


-0. 0004G2 


-0. 000462 


1.000000 


..a£ 






e. 185728 


0. 184857 


0. 999994 


:as 




J 


-0. 0080&2 


-0.013225 


0. 999832 


laz 




m 


-a. 187385 


-0. 186776 


0. 999827 


ia& 






-0. 1 88005 


-0. 167819 


0. 999863 




2 


« 


-0. 17430A 


-0. 172579 


0. 999979 




2 


b 


-0. 101 067 


-0. 100631 


0. 999999 


lea 


2 


c 


-0. 137425 


-0. 136297 


0.999991 




2 


d 


-0. 223362 


-0. 221788 


0. 999982 


182 


2 


• 


-0. 037327 


-0. 037327 


1 . 000000 




2 




-0. : 47249 


-0. 146781 


0. 999998 


las 


2 


c 


-0. 195747 


-0. 194583 


0. 999990 


ia£ 


2 


h 


0. 188673 


0. 184829 


0. 999864 




2 


J 


-0. 009834 


-0. 018578 


0. 999581 


182 


2 


I 


0. 157129 


0. 155412 


0. 999968 


laz 


2 




-0. 198272 


-0. 197672 


0. 999900 


182 


2 




-0. 190478 


-0. 192268 


0. 999884 



p > .05 



-1.386175 
-1.021391 

0. 000000 

1.386589 
1.491479 
-0. 170309 
0. 587933 
-1.414123 
-1.424118 
-1.417226 
-1.432104 

0. 000000 

-1.400394 
-1.385283 
1.251626 
1.598529 
1. 140827 
-0. 226425 
0. 633289 



NOTES: 

Q: Queries 100>199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 > Cosine. #2 « Dice 
TH: TeriR Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's Judgments were scaled from low to high (1 « most relevant, 
4 ■ most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance Judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (riy.> rjy). If this Z Is statistically 
significant as Indicated by tnc asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



INDEFINITES 



Correlation Coefficients 



Significance Level 



TW 



ifl* 1 • -0. 132394 

lOA 1 d 0. 129099 

184 1 • 0.064S21 

184 1 M -0.045491 

^84 1 J 0.003093 

184 1 m 0.080914 

184 1 n -0.007330 

184 2 a -0.148937 

-84 2 b -0. 1391S5 

184 2 c -0. 145872 

184 2 d 0. 128770 

184 2 • 0.142193 

284 2 f 0.039625 

184 2 g 0.121064 

184 2 h -0.032087 

*84 2 J 0.038815 

184 2 I -0.004347 

184 2 m 0.076227 

184 2 'n -0.024803 - 



-0. 132581 
0. 127115 
0. 062900 
-0. 045629 

0. 006609 
0. 080696 
-0. 003656 
-0. 149158 
-0. 139231 
-0. 146048 
0. 128098 
0. 140976 
0. 039648 
0. 120505 
-0- 034065 
0. 039566 
-0. 004624 
0.076297 
9. 023155 



ur 

0. 999986 
0. 999601 
0. 999970 
0. 999958 

0. 998948 
0. 998909 
0. 999516 
0. 999987 
0. 999998 
0. 999990 
0. 999892 
0. 999974 
0. 999976 
0. 999914 
0. 999303 
0. 999590 
0. 999980 
0. 999811 
0. 999903 



P > .05 



0. 160857 
0. 316625 
0. 768734 
0. 067423 

-0. 342742 
0. 020933 
-0. 527994 
0. 195327 
0. 166794 
0. 173707 
0. 206697 
0. 763283 

-0. 014669 
0. 192686 
0. 237118 

-0. 117423 
0. 195383 

-0. 016219 

-0. 527956 



NOTES: 

Q: Queries 100-198 were searched on INSPEC: 200-299 on PsychlNFO 
S: SiRllarUy Measure: II « Cosine. 12 « Dice 
TH: Terii Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance judgment and the 
systciR's predicted relevance based on unresolved anaphors. rjf Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 • most relevant, 
4 • most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rir> rju). If this Z Is statistically 
significant as indicated by tne asterisks, then resolving anaphors Improves 
the system's predicatlow of relevance. 

216 
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Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



RESIDUAL ADJECTIVES 



Correlation Coefficients Significance Level 



y 


c 


TW r. 




"ur 


7 
















10: 


- 


a 


-0. 424357 


-0.422512 


0. 999964 


-1. 198307 


101 




d 


-0. 745425 


-0. 751612 


0. 999041 


1.047436 


101 




• 


-0. 338200 


-0. 337852 


0. 999810 


-0. 096571 


101 


] 


h 


-0. 084690 


-0. 079805 


0. 999592 


-0. 874708 


101 


•1 


J 


-0. O05974 


-0. 561823 


0. 988776 


-1.741399 


101 


1 


m 


-0. 742554 


-0. 746668 


0. 998558 


0.579180 


101 


1 


ri 


-0. 692001 


-0. 698717 


0. 998276 


0. 797335 


101 


& 


* 


-0. 381796 


-0. 379436 


0. 999941 


-1. 188199 


101 




b 


-0. 328640 


-0. 327643 


0. 999988 


-1.289199 


101 




c 


-0. 381263 


-0. 379214 


0. 999955 


-1. 185281 


101 




d 


-0. 701048 


-0. 700325 


0. 999297 


-0. 137691 


101 




• 


-0.339941 


-0. 339150 


0. 999428 


-0. 126865 


101 




f 


-0. 662621 


-0. 662366 


0. 998983 


-0. 038515 


101 




S 


-0. 701503 


-0. 700533 


0. 999215 


-0. 175042 


101 




h 


-0.031633 


-0. 031677 


0. 999999 


0. 144143 


101 


z 


J 


-0.712239 


-0. 707026 


0. 998583 


-0. 696802 


101 


2 




-0.001532 


-0.001532 


1 . 000000 


0. 000000 


i01 


2 


Tfi 


-0. 731639 


-0. 729278 


0. 998978 


-0. 386272 


101 


2 


h 


-0. 741866 


-0.741931 


0. 998306 


0. 006552 



P > .05 



NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 * Cosine. 12 « Dice 
TW: Term Mtlghting Schemes: See Result Page R-1 

Correlation Coefficients: r^u Is between the user's relevance Judgment and the 
iystim*t predicted relevance based on unresolved anaphors. rw Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because tht user's judgments were scaled from low to high (1 « most relevant. 
4 ■ most non-relevant) a strono negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z indicates that the second correlation Is higher 
than the first correlation (r4r> rju). If this Z is statistically 
significant as Indicated bgr tne asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 

RESIDUAL ADJECTIVES 

Correlation Coefficients Significance Level 



Q 


s 


TM r. 




r 


z 












ur 






1 


a 






0« 339353 


0. 665002 




•• 




-0. 05399£ 


-0. 051487 


0. 999946 


-1. 182141 




4 

X 


m 


-0. 0005£8 


-0. 000S£8 


1 . 000000 


0. 000000 


:03 


4 


n 


-0. 319705 


-0. 3£0319 


0. 99998£ 


0. 527310 


-.03 






-0.3.16556 


-0. 3£69ia 


0. 996357 


0. 626573 


103 


4 


rf» 


-0. 333513 


-0. 3£7510 


0. 999£60 


-0.817553 


103 


X 


ri 


-0. 3£71£9 


-0. 3£1477 


0. 999546 


-0. 965824 


103 


£ 


a 


0. £57744 


0. £56450 


0. 999941 


0. 602435 


103 




b 


0. £95£30 


0. £94689 


0. 999988 


0. 563265 


103 




c 


0. £56569 


0. £55414 


0. 999951 


0. 592566 


-.03 


£ 


d 


-0. 0037£6 


-0. 003883 


0. 99998£ 


0. 126814 


103 


£ 




~0. 0003£6 


-0. 000528 


1 . 000000 


0. 000000 


103 


£ 


f 


0. 03£8S6 


0. 03£771 


0. 999996 


0. 142771 


103 


£ 


c 


-0. 006356 


-0. 008464 


0. 999984 


0. 091623 


103 


£ 




-0. 314465 


-0.315726 


0. 999959 


0.717689 


-.03 




J 


-0. 350808 


-0. 356913 


0. 9983£9 


0. 551869 


103 


£ 


1 


-0. £49678 


-0. £49491 


0. 999991 


-0. 220407 


2 03 


£ 


m 
• 


-0. £13815 


-0. £13207 


0. 999812 


-0. 157101 


103 


£ 




-0. £4£461 


-0. £42560 


0. 999784 


0. 0241 93' 



P > .05 



NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 « Cosine. 12 « Dice 
TU: Term Weighting Schemes: See Result Page R-l 

Correlation Coefficients: r^y Is between the user's relevance Judgment and the 
system's predicted rtmance based on unresolved anaphors. rw Is between 
the uMr*s rtlevanct judgment and the system's predicted relevance based 
on rtsolvid anaphors. 

Because the user's Judgments were scaled from low to high (1 ■ most relevant, 
4 > most non-relevant) a strono negative correlation shows agreement 
between user's and system's relevance Judgments. 

Signlflcanct Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rir> rjy). If this Z Is sutlstlcally 
significant as Indicated by the asterisks, then resolving anaphors Improves 
the liystea's predications of relevance. 21.8 
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Page 

A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



RESIDUAL ADJECTIVES 



Correlation Coefficients 



Significance Level 



TU 



ju 



ur 



p > .05 



194 






-0. 0712S2 


-0.071204 


0. 999904 


-0. 018778 


X0A 




d 


-0. £23340 


-0. 623721 


0.999985 


0. 4i»14a2 


104 


1 


• 


-0.429S65 


-0. 430780 


0. 999996 


1.324326 


104 


A- 


h 


-0. 347650 


-0. 347405 


0.999997 


-0. 486 1S2 


104 




J 


-0. 470712 


-0. 467840 


0. 999363 


-0. 406482 


134 




m 


-0. 6021S9 


-0. 600803 


0. 999930 


-0. 636572. 


;04 


4 


n 


-0. 415472 


-0. 383766 


0. 994267 


—1.410728 


1(»4 




a 


-0. 153381 


-0. 154276 


0.999859 


0. 080141 


Z04 


£ 


b 


-0. I&l9e0 


-0. 162270 


0.999969 


0. 165467 


1«4 


£: 


e 


-0. 149594 


-0. 149793 


0. 999884 


0. 059193 


;04 




c 


-0.592005 


-0. 592818 


0.999973 


0. 505450 


104 


£ 




-0.42S036 


-0. 427124 


0. 999991 


1.241918 


104 






-0.577165 


-0. 577596 


0.999993 


0. 633143 


104 






-0. 595353 


-0. 596113 


0.999976 


0. 600343 


1«4 


£ 


t% 


-0. 178700 


-0. 179124 


0. 999996 


0. 679858 


i04 




2 


-0. 472299 


-0. 475148 


0. 999825 


0. 766603 


104 






-0. 017455 


-0.017455 


1. 000000 


0. 000000 


'.04 






-0. 565612 


-0. 565380 


0.999964 


-0. 148670 


104 




n 


-0. 384919 


-0. 356937 


0. 995316 


-1. 366158 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNPO 
S: Similarity Measure: #1 « Cosine. #2 « Dice 
TV: Term Melghtlng SdMmes: See Result Page R-1 

Corrtlatloii CBcfflclcnts: rju is between the user's relevance judgment and the 

SttSi's predicted relevance based on unresolved anaphors. rjr is between 
t uttr's relevance Judgment and the system's predicted relevance based 
on resolved anaphors. 

lecauM the user's judgments were scaled from low to Mgh (1 » most relevant* 
4 • most non-relevant) a strong negative correlation shows agreement 
bet w aen user's and system's relevance Judgments. 

SIgnlffcanet Level: A positive Z Indicates that the second correlation Is higher 
then the first correlation ir*r> rju). If this Z Is sutlstlcally 
sfgnfffcant as Indicated ty tne asterisks* then resolving anaphors Impreves 
the lystam's prwHoitlons of relevance. 
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A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgacnts: for Anaphoric Class 

RESIDUAL ADJECTIVES 



Correlation Coefficients 



Significance level 



Q 


s 


T1 


■•■'Of 




1^ 






d 


1«7 


< 


». 


107 


1 


n 


107 


ji 


J 


107 




m 


.107 


r 


n 


107 




a 


.107 






107 




c 


t0T 




a 


107 




m 


107 




f 


107 






V^07 




fi 


107 




J 


107 




1 


X07 






107 


a 





-0. 
-0. 
-0. 

0. 

-0. 



-0. 
-0. 
-0. 

-0i 

-0. 
-0. 
-0. 

0. 
-0. 

0. 
-0. 
-0. 



26S170 -0. 
3SI714 -0. 
00i0SS -0. 
152533 0. 

185056 H9. 
327307 -0. 
Sa393£ -0. 
290729 -0. 
•285007 
292260 -0. 
35SSt7 
001095 -0. 
357481 -0. 
556079 -0. 
18239A 0. 
089937 -0. 
074758 0. 
315773 -0. 
296199 -0. 



■•jr 



269170 
3660eS 
001095 
152809 
187224 
331679 
288712 
290729 
285007 
292260 
358899 
001095 
358801 
358155 
181926 
093350 
074394 
318551 
299772; 



ur 

1^000000 
0.999845 

1.000000 

0.999998 
0.999938 
e. 999883 
0.999849 

• wVWW 



P > .05 



1.000000 
0.999966 
1.000000 
0.999988 
0.999972 
0.999988 
0.999861 
0.999998 
0.999941 
0.999901 



0. 000000 
1. 064491 
0.000000 
-0. 657191 
0.815546 
1.231026 
1. 171721 
0.000000 
0.000000 
O* 009000 
1.199868 
0.000000 
1. 194389 
1. 198332 
0. 401789 
0. 846109 
0. 696253 
1.099238 
1.084805 



NOTES: 

Q: Queries 100-199 were starched on INSPEC: 200>299 on PsiychlNFO 
S: Sinllarity Neasurt: #1 » Cosint. #2 » Dice 
TU: Ttni Ktighting SdMMs: Stt Htsult Page R-1 

Corrtlatlon Ctotfflcitnts: r«y Is bttuttn the ustr's relevance judgnent and the 
systta's prtdlctad rtlttaoct testd on unrcsolvtd anaphors. rjr is bttwttn 
tht ustr's rtltvmct judgMRt and tht systca's prtdicttd rtltvanct bastd 
OR rtsolvtd ampliors. 

Btctusc tte MStr's Judgwnts wtrt sctltd fro* lew to high (1 MSt rtltvant, 
4 « Mst MNHrtltvaiit) a strong ntgative correlation shows agrtawn t 
bt tw t w user's and systt«*s reltvtnce judgwnts. 

Sligniflcaiice Ltvel: A positive Z indicates tlitt tht stcotd corrtlatlon is hightr 
thBe the first correlatioe (rfr> rjy). If this Z is statistically 
significaet at indicated tSt tsttrisfcs, thtn rtsolving anaphors iiyrevts 
the iQfstfli*! predicatieoi ef relevaect. 
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Page 

A Statistical Coaparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgaents with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 



RESIDUAL ADJECTIVES 

( 

Correlation Coefficients 



Significance Level 



TM 



Ju 



109 








103 




109 




109 


1 


109 


♦ 


109 




109 




109 


2 


109 




109 


2 


109 




109 




109 




109 


2 


109 


2 


109 


2 


109 


2 


109 


2 



n 
a 

3 

c 
c 

f 
S 

J 

"» 

A. 
•M 

n 



-0. £20519 
-0. 350544 
-0.000545 
-0.06604S 

-0. 131452 
-0.328732 
-0.358384 
-0.228134 
-0- 233510 
-0.234593 
-0. 285177 
-0.000545 
-0.293837 
-0.289566 
-0.068850 

-0. 125951 
-0.069214 
-0. 296651 
-0. 330370 



-0. 220519 
-0.360341 
-0.000545 
-0.066833 

-0. 115245 
-0. 307707 
-0.327905 
-0.228134 
-0. 233510 
-0.234593 
-0.285164 
-0.000545 
-0. 293866 
-0.289553 
-0. 069122 

-0. 116716 
-0.069945 
-0. 280523 
-0.303095 



ur 

1.000000 

0. 999978 

1.000000 

0.999878 

0.998777 

0. 998562 

0.996556 

1.000000 

1.000000 

1.000000 

0.999990 

1.000000 

0.999997 

0.999992 

0.999959 

0.999744 

0.998914 

0. 999037 

0.997139 



P > .05 



0.000000 
-0. 176233 

0.000000 

0. 271866 
-1.775086 
-2. 172109 <-«HH>» 
-2.051845 C«»«* 

0.000000 

0.000000 

0.000000 
-0.016233 

0.000000 

0.069221 
-O. 017410 

0.162325 
-2.204416 <♦♦♦♦ 

0.084582 
-2.030152 (•♦♦♦ 

-2.004619 <<MHH» 



NOTES: 

Q: QueHes 100-199 were searched on INSPCC: 200-299 on PsychlNFO 
S: Sfailarity Neasure: #1 * Cdslne. #2 » Dice 
TV: Teni Melghtfng Sdms: See Result Page R-1 

CanrelatfoR CoefflclcnU: t*» Is between the user's relevance judgasnt and the 
systea's predicted reletancc based on unresolved anaphors. rw is between 
the user's relevance Ju dj o an t and the systca's predicted relevance based 
oa resolved anaphors. 

lecausc the user's J udj a t nts were scaled froa low to high (1 » aost relevant* 
4 » aost non-relevant) a strong negative correlation shows agrcmnt 
between user's and sjfstea's relevance Judguents. 

Sfgnfffcanco Level: A posltfvt Z Indicates that the second correlation Is higher 
than the first correlation (r4r> rj„). If this Z Is sutlstlcatly 
significant as Indicated ly tie asterisks* than resolving aiaphors Inproves 
tbe lystaa's prodlcatlonK of relevance. 
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A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgaents: for Anaphoric Class 



RESIDUAL ADJECTIVES 



Correlation Coefficients 



Significance Level 



TH 
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a 


135 


1 




235 


1 


• 


135 


t 




135 






135 


1 


nn 


135 


1 


n 


135 




a 


135 




o 


135 




c 


135 




d 


135 




• 


135 




f 


135 




G 


135 






135 


2 


t 


135 


2 


I 


:35 


2 


HI 


135 


2 


h 



-o. 

-0. 

-e. 

-0. 

-0. 



590634 
79723S 
001234 
026503 
633372 
736876 
342132 
637630 
647033 
635240 
818316 
001234 
818360 
81618S 
001788 
766710 
001323 
313643 
826860 



-0.600304 
-0.736253 
-0.001234 
-0.035626 
-0. 644212 
-0.800067 
-0.811076 
-0.648353 
-0.652380 
-0.644584 
-0.813110 
-0.001234 
-0.818326 
-0.817176 
-0. 002120 
-0. 778223 
-0.001332 
-0.816604 
-0. 737563 



ur 

0. 333613 
0.333366 
1.000000 
0.338308 
0.360261 
0.3337S6 
0.335357 
0.333258 
0.333766 
0.393466 
0.333765 
1.000000 
0.333318 
0.333821 
0.393399 
0.335613 
1.000000 
0.393662 
0.335690 



P > .05 



1.704237 
0.806227 

0.000000 

. 0.805472 
0. 276614 

0. 876231 
-1. 829803 

1. 390371 

1. 247222 
1.418114 
0. 261931 
0.000000 

-0. 019273 
0. 370452 
1. 315987 
0.773603 
0.479305 
0. 781532 

->1. 778178 



NOTES: 

Q: Queries 100>199 were searched on INSPEC: 200-299 on PsychlNFO 
S: SMIaHty Neaswre: #1 « Cosine. #2 « Dice 
TM: Tewi Weighting 5r1iwy; See Result Page R-l 

Correlatfoii Oocfffcfcnts: rjg is between the user's relevance judgMnt and the 
Siystea's predicted releVaiict based on unresolved anaphors. rjr Is between 
the user's relevance iudgamt and the sjrstcn's predicted relevance based 
M resolved anaphors. 

lecausc the user's JydgMnU were scaled fraa low to high (1 « nost relevant, 
4 « wst uou-relevaat) a strong negative correlation shows agrecaent 
b e t w a e n user's and sjfstea's relevance Judguents. 

Significance Level: A positive X Indicates that the second correlation Is higher 
than the first correlatloii {x\t> ^jh). If tMs Z Is sutlstlcally 
sl^lflcant as Indicated by the asterisks, then resolving anaphors leprovcs 
the sjfstai's predlcatloai of relevance. 
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Page 

A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgaents: for Anaphoric Class 



RESIDUAL ADJECTIVES 



Correlation Coefficients 



ur 

0.999963 
0.999885 
1.000000 
0.780841 

0. 8S4939 
0.954047 
0.938905 
0.999945 
0.999992 
0.999961 
0.999884 
1.00OOOO 
0.999974 
0.999907 
0. 120421 

0.799655 
O. 395597 
0.959040 
O. 943510 
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142 


1 


a 


-0. 192139 


-0.1895:^ 


X42 


1 


d 


-0. 255597 


-0.257921 


142 


1 


• 


-0.000662 


-0.O0O662 


1*2 


1 


h 


-0.277455 


-O. 209714 


142 


I 


J 


-O. 318605 


-0.290542 


142 


1 


m 


-O. 304107 


-0.243396 


X42 


1 


n 


-0.211922 


-0.112039 


142 


2 


a 


-0. 324310 


-0.321402 


*42 


2 


b 


-O. 435620 


-0.434607 


142 


2 


e 


-0.355980 


-0.353568 


142 


2 


d 


-0.339456 


-0.340976 


142 


2 


• 


-O.O0O662 


-e. 000662 


142 


2 




-O. 452453 


-O. 453093 


142 


2 




-O. 352275 


-0.353638 


142 


2 


n 


-0.277419 


-0.027069 


142 


2 


J 


-0. 3X6247 


-0.335038 


142 


2 




-0.255325 


-0.001435 


142 


2 


• 


-0.359307 


-0. 321676 


142 


2 


n 


-0.254817 


-O. 194305 



Significance Level 



-1.337336 
O. 690355 
O. 0 00000 
-O. 461590 
-O. 218007 
-O. 905335 
-1.263212 
-1.265263 
-1.209647 
-1.251661 
O. 462161 
0.000000 
O. 429176 
O. 464213 
-O. 853365 

O. 137673 
-1.038994 
-O. 608619 
-O. 804963 



p > .05 



NOTES: 

Q: Queries 100->199 were searched on INSPEC: 200>299 on PsychlNFO 
S: SMIarity Neasuro: #1 « Cosine. §1 « Dice 
TV: Teni Kelghtlng SdMMs: See Result Page R-1 

Corrclatloii Coefficients: rjy Is between the user's relevance Judgnent and the 
systca's predicted rcletance based on unresolved anaphors. rjr Is between 
the user's relevance Judgnent and the sjfstcn's predicted relevance based 
on resolved antpiiors. 

Because the user's Judgnents were scaled fron low to high (1 « nost relevant, 
4 • nost non-relevant) a strong negative correlation shows agrccncnt 
b e t w een user's and sjrstea's relevance Judgnents. 

Signlflcanct Level: A positive Z Indicates that the second correlation Is higher 
then the first correlation (r«r> rju). If this Z Is sUtlstlcally 
significant as Indicated by tee asterisks, then resolving amphors Inproves 
the lystfli's predications of relevance. 
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Page 

A Statistical Coaparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 

RESIDUAL ADJECTIVES 



Correlation Coefficients 



Significance level 



156 

158 

158 

258 

158 

158 

158 

158 

158 

158 

158 

158 

158 

158 

158 

158 

158 

158 

158 



1 
1 
1 



1 
1 

2 
2 
2 
2 
2 
2 
2 
2 



• e. 

c -0, 

• -0. 

h -0. 

J 

m -0, 
n -0. 

• 0. 

0 0. 
C 0. 

a -0. 

• -0. 
f 0. 

9 -0. 
.*» -0. 

1 -0. 
I -0. 
n -0. 
n -0, 



036318 

102911 

001032 

2£1133 

172647 

212637 

190370 

090669 

174475 

116130 

113967 

001032 

016314 

069306 

136551 

103790 

007262 

131206 

120776 



0. 037274 
-0. 164549 
-0.001032 
-0.266308 
-0. 167373 
-0. 221337 
-0.2106S2 
0. 069326 
0. 173910 
0. 115103 
-0. 113303 
-0.001032 
0. 016700 
-0.066465 
-0. 175042 
-0, 119243 
-0.020390 
-0. 137477 
-0. 136421 



ur 


z 


P > 


0. 999661 


0. 234405 




0.999945 


0. 593286 




1.000000 


0.000000 




0.999643 


1.034222 




0. 991239 


0. 417235 




0. 999757 


1.496619 




0.992794 


0. 644254 




0.999609 


0.257476 




0.999972 


0. 285531 




0.999675 


a. 244216 




0.999831 


-0. 136053 




1.000000 


0.000000 




0.999829 


-0. 076035 




0.999636 


-0. 174200 




0.997408 


2.004695 


<«««» 


0. 996767 


1. l^^d 




0.999677 


1.932696 




0.999637 


1.307669 




0.996092 


0.667111 





NOTES: 

Q: (hicrles 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Slallarlty Measure: #1 - Cosine. f 2 » Dice 
TV: Tens Helgbtlng SdwMs: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance JudgnMit and the 
systea's predicted reletance based on unresolved anaphors. rjr Is between 
the user's relevance judgMnt and the systai's predicted relevance based 
on resolved anaphors. 

Secause tfce user's Judjunts were scaled fro* low to high (I » nost relevant, 
4 • nost non-relevant) a strong negative correlation shows agreoMnt 
between user's and systea's reievtnce JudgMnts. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation {x\t> rjy). If this Z is sutlstfcally 
significant as Indicated Igr the asterfsks, then resolving amphors Inproves 
the vstea's predications of relevance. 
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Page 

A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 

RES IDUAL ADJECTIVES 



Correlation Coefficients 



Significance Level 



TW 



I7fil 
i70 
170 
17© 
170 
170 

:70 

170 
170 
170 
;70 
170 
170 
170 
170 
170 
170 
170 
170 



1 
1 
1 

4 

<*■ 

1 
1 
1 
2 
& 
2 

a 

S 
2 

a 

2 
2 

2 



h 

m 
n 
a 

b 
c 
d 

f 

8 

n 



m 



-0.716613 
-0.731051 
-0.001109 
0. 166612 
-0. 594562 
-0.636259 
-0. 515708 
-0. 678385 
-0. 637558 
-0.678090 
-0. 691708 

-0. 001109 

-0. 696939 
-0.689511 

0. 153761 
-0. 553947 

0.045678 
-0. 623943 
-0. 533728 



-0. 716201 
-0. 742090 

-0.001109 

0. 164412 
-0. 614372 
-0. 654478 
-0. 541451 
-0. 6781 12 
-0. 637414 
-0.677830 
-0. 707218 
-O. 001109 
-0. 713767 
-0.705860 

0.152558 
-0. 575755 

0.045717 
-0. 644082 
-0. 560072 





7 


^r 




0. 999967 




0. 993649 




1.000000 


0.000000 


0. 999960 


1.089657 


0. 990243 


0. 764024 


0. 992729 


0. 8429S3 


0. 991535 


0. 994660 


0. 999979 


-0. 249519 


0.999996 


-0. 274505 


0. 999983 


-0. 262757 


0. 993231 


0. 794390 


1.000000 


0.000000 


0. 989529 


0. 703046 


0. 992551 


0.796411 


0. 999978 


0. 601444 


0. 988949 


0. 764538 


0.999997 


-0. 067077 


0. 991707 


0.861803 


0. 990025 


0. 951460 



P > .05 



NOTES: 

Q: Queries 100>199 were searched on INSPEC: 200-299 on PsychlNFO 
S: SiMflarlty Measure: #1 « Cosine. #2 • Dice 
TU:. Terw Weighting SdMMs: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance judgeent and the 
Sjfsten's predicted reletance based on unresolved anaphors. rjr Is between 
the user*s relevance Judgwnt and the systen's predicted relevance based 
on resolved anaphors. 

Because the user's JudgMnts were scaled froa low to high (1 • nost relevant, 
4 ■ nst non-relevant) a strong negative correlation shows agreeNnt 
between user's and sjrsteai's relevance JudgMnts. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (r4r> rju)* ^ 1* sutlstlully 

significant as Indicated Igr tne asterisks, then resolving anaphors inproves 
tbt predications ef relevance. 
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Page 

A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 

RESIDUAL ADJECTIVES 



Correction Coefficients 



Significance Level 
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a -0. 
d -0. 

• -9. 
h -0. 

J -0. 

M -0. 

n -0. 
a -0. 
b -0. 
c -0, 
ct -0. 

• -0. 

f -0. 

0 -0. 

h -0. 

J -0. 

1 -0. 
m -0. 
n -0. 



■•ju 



297536 
475084 
484654 
234297 
395090 
473372 
428501 
219548 
157462 
213261 
475058 
434632 
393963 
468597 
352302 

449142 
126480 
476596 
445123 



-0. 300069 
-0. 469676 
-0. 484654 
-0. 250499 
-0. 391709 
-0. 469709 
-0. 426252 
-0. 223547 
-0. 159451 
-0.216810 
-0. 463184 
-0. 434632 
-0. 372357 
-0.454645 
-0. 377455 

-0.447423 
-0. 138939 
-0. 473019 
-0.445674 



ur 

0. 999718 
0. 9975S1 
1.000000 

0.998062 
0. 998332 
0. 998407 
0.999451 
0.999558 
0.999902 
0. 999676 
0. 996392 
000000 
0.995482 
0. 996089 
0.991003 

0. 999231 
0. 999040 
0. 998716 
0. 999652 



P > .05 



0. 556162 
-0. 438159 

0. 000000 

1.332640 
-0. 316095 
-0. 367378 
-0. 374795 
0. 689100 
0. 719927 
0. 713390 
-0. 785163 
0.000000 
-1.214745 
-0.666267 
0.999521 

-0.245109 
1.431369 

-0. 400072 
0.116776 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200>299 on PsychlNFO 
S: SlRrilarlty Measure: #1 ■ Cosine. #2 » Dice 
TW: Teni Nclghtlng SchcMs: Sec Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance Judgmnt and the 
^stea's predicted rcletance based on unresolved anaphors. rjr Is between 
the user's relevance judgMnt and the system's predicted relevance based 
on resolved anaphors. 

Beciust the user's judgMnts were scaled fro« low to high (1 « aost relevant. 
4 • «tt non-relevant) a strong negative correlation shows agrecMnt 
betMeen user's and systen's relevance JudgMnts. 

Significance Level: A posltfvt Z Indicates that the second correlation Is higher 
than the first correlation (r4,.> rj„). If this Z Is sUtlstlcally 
significant as Indicated ly the asterisks, then resolving anaphors Inproves 
the ^tea's predications of relevance. 
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Page 

A Statistical Coaparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



RESIDUAL ADJECTIVES 



Correlation Coefficients 



Significance Level 
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P > .05 
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0. 994889 


-1.367066 
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-0. 161261 


0. 992680 


-1. 294802 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Slallarlty Measure: #1 * Cosine. 12 • Dice 
TU: Tera Weighting Schenes: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance Judgmnt and the 
systcn's predicted relevance based on unresolved anaphors. rj,. Is between 
the user*s relevance judgnent and the system's predicted relevance based 
on resolved anaphors. 

Because the user's JudgMnts were scaled free low to high (1 * «ost relevant, 
4 « iBSt non-relevant) a strong negative correlation shows agrecMnt 
between user's and systea's relevance JudgMnts. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rjr> riy). If this Z Is sutlstlcally 
significant as Indicated fcy the asterisks, then resolving anaphors laproves 
the ^teM's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 

RESIDUAL ADJECTIVES 



Correlation Coefficients 



Significance Level 



TH 
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284 
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184 
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184 
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184 
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a 
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-0. 132394 
9. 129099 
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-0.045491 
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0. 080914 
-0. 007330 
-0. 148937 
-0. 139155 
-0. 145872 
0. 128770 
0. 142193 
0. 039625 
0. 121064 
0. 032087 
0. 038815 
-0. 004347 
0. 076227 
0. 024803 



-0. 132581 
0. 136537 
0.064221 

-0. 045478 

0. 008376 
0. 087276 
-0. 002087 
-0. 149158 
-0. 139231 
-0. 146046 
0. 133600 
0. 142193 
0. 042174 
0. 125731 
-0. 033874 
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-0. 004624 
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-0. 006535 
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0.999714 


-1. 193218 


0. 999859 


-1.396917 


0.999987 


0. 195327 


0.999998 


0. 166794 


0.999990 


0. 173707 


0.999816 


-1. 133947 


1.000000 


0. 000000 


0.999919 


. -0.897806 


0.999819 


-1. 103698 


0.999309 


0.215014 


0. 999831 


-0.712119 


0. 999960 


0. 195383 


0. 999855 


-0. 982313 


0. 999962 


-1. 138266 



P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 - COslne. #2 « Dice 
TU: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance Judgment and the system's predicted relevance based 
Oft resolved anaphors. 

Because the user's Judgments were scaled from low to high (1 • most relevant* 
4 « most non-relevant) a strono negative correlation shows agreoMnt 
bet ween user's and system's relevance Judgments. 

Signlflcanct Level: A positive Z Indicates that the second correlation Is higher 
thin the first correlation (r4|.> rjy). If this Z is statistically 
sfgnff leant as Indicated by the asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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Page 

A Stetisticil Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgiients: fo? Anaphoric Class 

ADVERBS 



Correlation Coefficients 



Significance Level 
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m -0, 
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69£001 
38X796 
3£8849 
381£83 
701048 
339941 
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-0. 4££51£ 
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-0. 4£0617 
-0. 07££47 
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-0. 740466 
-0. 677530 
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-0. 3£7643 
-0. 379£14 
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0.999955 
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0.960£34 
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0. 998551 
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0. 998631 
0. 997741 



P > .05 



-1. 198307 
0.916500 
1.433873 

-1.5930£2 

-1. 8£77££ 
-0. £39187 
-0. 785997 
-1. 188190 
-1. £89199 
-1. 185£8l 
1.197970 
1.424931 
1. 173007 
1. 184736 
-0. ££6240 
-0. 166544 
0.000000 
0.619011 
-0. 6£6018 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNPO 
S: Similarity Measure: #1 » Cbslne. #2 » Dice 
TH: Teni Weighting ScheMes: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance Judgment and the 
system's predicted relcVance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 • most relevant* 
4 - most nonnrelevant) a strong negative correlation shows agreflwnt 
between user's and system's relevance Judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rj,.> r4„). If this 2 Is sutlstlcally 
significant as Indicated by the asterisks, then resolving anaphors Improves 
the lystM's predications of relevance. ^ 
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Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 



Correlation Coefficients 



Significance Level 
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0. 255414 
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0.999993 


-0. 822045 


0. 997681 


0. 943716 


1.000000 


-1. 329759 


0. 994566 


1.061138 


0. 993529 


0. 923036 



P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: SlallaHty Heasure: #1 - COslne. #2 - Dice 
TN: Tem Weighting Schcaes: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance judgmnt and the 
systcn's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgsent and the systCM's predicted relevance based 
on resolved anaphors. 

Because the user's Judgaents were scaled fro« low to high (1 « aost relevant, 
4 ■ Most non-relevant) a strong negative correlation shows agreeaent 
between user's and syste«*s relevance JudgMnts. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation in^^ rju). If this Z Is statistically 
significant as Indicated ty the asterisks, then resolving anaphors Inproves 
the lystai's predications of relevance. 
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Pege 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



ADYERBS 



Correlation Coefficients 



Significance Level 
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-0. 149793 
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0.999859 
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0.999969 
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0. 999884 
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0. 999977 
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1.281626 


0. 997362 


1.379610 


0.999998 


1.079241 


S. 997540 


1. 201384 


0. 988233 


-0. 244910 



P > .05 



NOTES: 

Q: Queries 100>199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 « Cosine. #2 « Dice 
TV: Term Weighting Scha^s: See Result Page K-l 

Correlation Coefficients: rjy Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved antphort. 

Because the user's Judgments uere scaled from low to high (1 » most relevant* 
4 • most nonnrelevant) « strong negative correlation shows agreement 
between user's and system's relevance JudgMnts. 

Significance Level: A positive Z Indicates thtt the second correlation Is higher 
then the first corrtUtlen (ri^ > rjy). If this Z Is sUtlstlcally 
significant as Indicated ly tie asterisks* then resolving anaphors teroves 
the nystam's predications of relevance. 
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Page 

f Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgaents with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 

APyERBS 



Correlation Coefficients 



Significance level 
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P > .05 



0.000000 
O. 462016 
0.OO0000 
0.8203S8 
1. 130729 
O. 537701 
O. 752152 
O.O€M9000 
0. 000000 
O.OOOOOO 
-0.130842 
0.000000 
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-O. 187974 
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1. 136444 
O. 891345 
0.206098 
0- 746373 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlHFO 
S: Slallarlty Measure: #1 » Cosine. #2 » Dice 
TV: Tenilfclglitliig SdMnes: See Result Page R-l 

Corrclatloo Goofffclcots: rj„ f$ between the user's relevance Judgnent and the 
system's predicted rtlciance hased on unresolved anaphors. rjr Is between 
^ ^IT^^'^f*^ JudgMnt and the systOM's predicted relevance based 
on resolvod anaphors. 

Because the user's JudgnenU were scaled fren low to high (1 • nost relevant. 
4-^t BonHrelcvant) a strong negative correlation shows agrewmt 
between user's and %y%tm's relevance Judgnents. 

SlgniflMnce^evel: A posltfve Z Indicates^ that the second correlation Is higher 
than the first corrolatlon Crj,.> rj„). If tMs 2 Is sUtlstlcally 

^JS*«*!^ ^ attwrlsks, then resolving anaphors Inproves 
the ^tOT's pmdicatlont of relevance. ^ 
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Page 

A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgaents with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 



Correlation Coefficients Significance Level 
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P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPCC: 200-299 on PsychlNFO 
S: Slallarlty fteasure: 11 « Cosine. #2 « Dice 
TN: Teni Nelghtli^ SdMMs: See Result Page R-1 

Csnrelatloii Co«ff1c1|||s: rjy Is between the user's relevance Judgnent and the 
systOMlf predlclR releVance based on unresolved anaphors. rjr Is between 
the user's rtlevanct judgMnt and the systOH's predicted relevance based 
4iNi resolved amphors. 

lecause the user's judgaents were scaled fro« low to high (1 » nost relevant, 
4 B wst non-relevant) a strong negative correlation shows agreencnt 
between user's and systeii's relevance judgwnts. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
thin the first correlation (ti^y rjy). If this Z Is sUtlstlcally 
significant as Indicated fey tie asterisks, then resolving anaphors Improves 
the Dfstfli's predications of relevance. 
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A Statf stfcti CoaparfsoR of the Relatlonshfp Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgiaents: for Anaphoric Class 
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Correlation Coefficients Significance Level 
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P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Slallarlty Measure: #1 - Cosine. iZ « Dice 
TH: Teni Melghting SdMMs: Sm Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance judgnent and the 
^stea's predicted relevance hased on unresolved anaphors. rir Is between 
the user's rtlevaact JudgMnt and the systca's predicted relevance based 
on resolved anaphors. 

Because the user's judyMnts were scaled froM low to high (1 » nost relevant. 
4 ■ matt non-relevant) a strong negative correlation shows agrecMnt 
between user's and sjrstea's relevance judgnents. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rj,.> rjy). If this 2 Is sUtlstlcally 
significant as Indicated ly tie asterisks, then resolving aniphors Inproves 
the vstea's predications of relevance. . 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgaents with Resolved 
Anaphors and User's Relevance Judgments : for Anaphoric Class 
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Correlation Coefficients Significance Level 
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2 


c 


-0.355960 




0. 999961 


-1.251661 


142 


2 


d 


-0.339456 


-0.339864 


0.999135 


0.047741 


142 


2 




-0.000662 


-0.000662 


1.000000 




142 


2 


f 


-0.452453 


-0.450467 


0.998652 


-0. 186777 


142 


2 


B 


-0.352275 


-0.351640 


0.998953 


-0.064632 


142 


2 


h 


-0. 277419 


-0.027069 


0.120421 


-0.853365 


142 


2 


J 


-0.316247 


-0.338458 


0.797386 


0. 161938 


142 


2 


I 


-0.255325 


-0.001435 


0.395597 


-1.038994 


142 




m 

* 


-0.359307 


-0. 322414 


0.957514 


-0.586120 


142 


2 


n 


-0.254817 


-0. 198763 


0.942636 


-0.740485 



P > .05 



ROTES: 

Q: queries 100-199 Mtre searched on INSPEC: 200-299 on PsychlNFO 
S: Slallarity Ntasure: #1 » Cosine. #2 « Dice 
TM: Teni HelghtlRf SchHMs: See Result Page R-1 

Oorrelatloii CoefffclinU: rjy Is betneiR the user's relevance jiidgoent and the 
«ystM*s predicted relevance hased on unresolved anaphors. Is betueen 
the user's relevance JudgMnt and the system's predicted relevance based 
on resolved anaphors. 

Because the user's iudgMnts uere scaled freo low to Mgh (1 « oost relevant. 
4 » wst non-relevant) a strone negative correlation shows agrecMnt 
between user's and system's relevance judgnents. 

Significance Level: A positive Z Indicates thnt the second correlation Is higher 
thin the first correlation (rjr> rjy). If this Z Is sutlstlcally 
significant as Indicated t» the asterisks, than resolving anaphors liyroves 
the Stan's predications of relevance. 
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Appendix E-92 

A Statistical Coaparisea of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgaents with Resolved 
Anaphors and User's Relevance Judgiients: for Anaphoric Class 



Correlation Coefficients 



ur 



Q 


S 




••jr 


158 


1 




0. 938318 


9. 937274 


isa 


1 


d 


-e. 162911 


-9. 167314 


158 


1 




158 


1 


h 


-0.261133 


-9.268266 


158 


1 


J 


-0. 172847 


-9. 186749 


158 


1 


Mr 


-8.212637 


-9.222111 


158 


1 


n 


190379 


-9.299259 


158 


2 


a 


8.990669 


9.989328 


158 




b 


0. 174475 


9. 173919 


158 


a 


c 


0.116139 


9.115193 


158 




d 


-9. 113967 


-9. 125949 


158 


2 




-0.991932 


-9.991932 


158 


a 




9.916314 


9.999695 


158 


a 


8 


-9.989396 


-9.192465 


158 


2 


h 


-9.136551 


-9. 174153 


158 


2 


J 


-9. 193799 


-9. 124319 


158 


2 


I 


-9.997262 


-9.929129 


158 


2 




-9. 131296 


-9. 146276 


158 


2 


• 
n 


-9. 129778 


-9. 136574 



Significance Level 



P > .05 



9.999861 


0.234405 


9.999524 


0.540739 




0.000000 


9.999657 


1.047817 


9.991273 


0.400075 


9.999715 


1.503621 


9.992913 


0.605052 


9.999899 


0.257478 


9.999972 


0.285531 


9.999875 


0.244216 


9.996291 


0.523739 


* • WWW 




9.996384 


0.687294 


9.996139 


0.562771 


9.997518 


2.001510 <»»«» 


9.998990 


1.248288 


0.999681 


1. 904769 


0.998073 


0. 915683 


0.996080 


0. 672644 ' 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: SiMilarity Measure: #1 « Cosine. #2 « Dice 
TV: Teni Meighting Sch«Ks: See Risult Page R-1 

CBrrelation GaefficicnU: rjy is between the user's relevance Judgnent and the 
SjfStiB's predicted rtletance hiscd on unresolved anaphors. rjr is between 
the* user's relevaact Judgatnt and the system's predicted relevance based 
on resolved anaphors. 

Because the user's Judgatnts ware scaled froa low to high (1 » aost relevant, 
4 • aost non-relevant) a strong negative correlation shows agrcoMnt 
between user's and sjfstaa's relevance Judg w nts. 

Significance Level: A positive Z indicates that the second correlation is higher 
than the first correlation (rj,.> rju). If this Z is sUtistically 
significant as indicated by the asterisks, then resolving anaphors inproves 
the lystai's predications of relevance. 
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Appendix £-93 



Page 

f SUtlstfctl Coaparisott of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgaents with Resolved 
Anaphors and User's Relevance Judgaents: for Anaphoric Class 



ADVERBS 



Correlation Coefficients Significance Level 

' ^ ™ "^ju ^Sr \r ^ P > -05 



179 


1 


a 


-0.716613 


-0.716201 


0.999987 


-0. 5O0158 


179 


1 


d 


731051 


-0.731320 


0.999647 


0. 064753 


179 


i 


• 


-«.e01ie9 


-0.001109 




0.000000 


179 


1 


h 


0. 166612 


0.165329 


0.999969 


0. 722759 


179 


t 


2 


-0.594562 


-0.595450 


0.998884 


0. 101955 


179 


1 


m 


-0.636259 


-0.635176 


0.999509 


-0. 195144 


179 


1 


n 


-0. 515708 


-0.512295 


0.999210 


-0.434523 


179 


2 


a 


-0.678385 


-0.678112 


0.999979 


-0.249519 


179 




b 


-0.637558 


-0.637414 


0.999996 


-0. 274505 


179 




c 


-0.678090 


-0.677830 


0.999983 


-0.262757 


179 




0 


-0.691708 


-0. 693355 


0.999701 


0. 404417 


179 




• 


-0.001109 


-0.001109 






179 


S 


f 


-0.696939 


-0. 697367 


999905 


0.186628 


179 




S 


-0.669511 


-0.691282 


0.999758 


0..480403 


179 




h 


0. 153761 


0. 153635 


0.999985 


0. 102754 


\79 


2 


J 


-0. 553947 


-0. 552176 


0.999329 


-0.252509 


179 


2 


I 


0. 045678 


0.046002 


0.999997 


-0. 622328 


179 


2 


. M 


-0. 623943 


-0.624003 


0.999621 


0.011997 


179 


2 


n 


-0. 533728 


-0.532179 


0.999460 


-0. 242661 



NOTES: 

Q: Queries 100-199 were seirched on INSPEC: 200-299 on PsychlNFO 
S: Siailarity Iteasure: #1 » Cosine. #2 « Dice 
TU: Tera Weighting Schcaes: See Result Page R-1 

Correlation Coefficients: rjy is between the user's relevance judywnt and the 
^stca's predicted rtlefance bised on unresolved anaphors. rjr is between 
the user's relevance judgaent and the systca's predicted relevance based 
en resolved anaphors. 

Because the user's judgacnU were scaled froa low to high (1 « aost relevant, 
4 • »st nonnrelcvant) a strong negative correlation shows agrc«Mnt 
between user's and sjfstea's relevance Judgaents. 

Significance Level: A positive 2 indicates that the second correlation is higher 

- * -f this 2 is sutistically 

then resolving anaphors inproves 



iiMiKv Level : a positive i, inncates tnat l 
than the first correlation (rj,.> rjy). If this 2 is sutistically 
significant as indicated tf the asterisks, 1 



the lystMi's predications of relevance. 
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Appendix E-94 



A Statlstictl Coaptrlson of the ReUtfonship Between 
Unresolved Anephors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgaents: for Anaphoric Class 



AWfERBS 



Correlation Coefficients 



Significance level 



TW 



ur 



P > .05 



180 


1 


• 


-0.297536 


-0.299409 


0. 999723 


230 


1 


a 


-0. 475084 


-0.465908 


0. 999376 


ise 


1 


9 


-0.484654 




X* 000000 


lae 


1 


h 


-0. 234297 


-0.250428 


0.998063 


lB^ 


1 


2 


-0.395090 


-0.396504 


0.996871 


180 


1 


m 


-0. 473372 


-0. 462114 


0.999679 


180 


1 


n 


428501 


-0.432483 


0.999846 


180 


a 


a 


-0.219548 


-:0. 222590 


0.999568 


j:a0 


2 


b 


-0. 157462 


-0.159036 


0.999904 


180 


Z 


c 


-0.213261 


-0. 215969 


0.999684 


180 


s 


d 


-0.475058 


-0. 487901 


0.999342 


180 




• 


-0.434632 


-0.434632 


* • WVMCFW 


180 




f 


-0.393963 


-0.404196 


0.999606 


180 




6 


-0.468597 


-0.481172 


0.999380 


180 


a 


h 


-0. 352302 


-0.377404 


0.991004 


180 


2 


J 


-0. 449142 


-0.454957 


0.999533 


180 


2 


I 


-0.126460 


-0. 136984 


0.999041 


180 


2 


W 


-0. 476596 


-0.486384 


0. 999647 


180 


2 


n 


-0.445123 


-0.450480 


0.999766 



0. 416233 

1.677809 

0.O0000O 

1.327498 

0.390883 

1.866891 

1.235124 

O. 530523 

O. 573887 

O. 554789 

1.914103 

0.000000 

1.916450 

1.925501 

0.99760O 

1.051901 
1.437095 
1.983032 
1.353539 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Slollarlty Measure: #1 « Ctoslne. #2 * Dice 
TW: Teni Weighting Schcaes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance judgnent and the 
systoa's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgaent and the systca's predicted relevance based 
on resolved anaphors. 

Because the user's judgaents were scaled froa low to high (1 » aost relevant, 
4 ■ nost non-relevant) a strong negative correlation shows agrccacnt 
between user's and systea's relevance judgaents. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rjr> rju). If this Z Is sutlstlcally 
significant as Indicated ly the asterisks, then resolving anaphors laproves 
the lystoi's predications of relevance. 
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Appendix E-95 

Page 

A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgaents with Resolved 
Anaphors and User's Relevance Judgaents: for Anaphoric Class 

ADVERBS 



Correlation Coefficients 



Significance Level 



ur 



P > .05 



1S2 


1 


* 


194413 


-0. 192936 


0.999964 


-1.386175 


182 


1 




-e. 221779 


-9, 224193 


0.996365 


0. 153693 


182 


1 


• 


-0.000462 


-0.000462 


!• 990009 




182 


1 


h 


0. 185728 


0. 185061 


0.999996 


1.301022 


182 


1 


J 




-0.005605 


0.998273 


-0. 203242 


182 


1 


m 


-0. 187365 


-0. 184546 


0.996166 


-0. 173346 


182 


1 


n 


-0. 166005 


-0. 166631 


0.997851 


0. 051283 


182 


2 


a 


-0. 174304 


-0. 172579 


0. 999979 


-1.414123 


182 


2 


b 


-0. 101067 


-0. 100631 


0.999999 


-1.424118 


182 


2 


c 


-0. 137425 


-0. 136297 


0.999991 


-1.417226 


182 


2 


d 


-0. 223362 


-0.221232 


0.994130 


-0. 106675 


182 


2 


• 


-0. 037327 


-0.037327 


1.000000 


V* WWW 


182 


2 


f 


-O. 147249 


-0. 147663 


0.998098 


0. 035948 


182 


2 


8 


-0. 195747 


-0. 194440 


0.995430 


-0. 073752 


182 


2 


n 


0. 188673 


0. 185455 


0.999907 


1. 267652 


182 


2 


J 


-0. 009834 


-0. 016492 


0.997645 


0. 513395 


182 


2 


1 


0. 157129 


0. 155712 


0.999978 


1. 131076 


182 


2 


M 


-0. 198272 


-0. 196727 


0.995693 


-0. 089856 


182 


2 




-0. 190476 


-0. 193120 


0. 998452 


0.256058 



NOTES: 

Q: Queries 100>199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Slallarlty Measure: #1 « Cosine. #2 - Dice 
TN: Tera Weighting Schcaes: See Result Page R-1 

Correlation Coefficients: r*^ Is between the user's relevance Judgaent and the 
systea's predicted relevance based on unresolved anaphors. rj^ Is between 
the user's relevance judgaent and the systea's predicted relevance based 
on resolved anaphors. 

Because the user's judgaents were scaled froa low to high (1 * aost relevant, 
4 ■ aost non-relevant) a strong negative correlation shows agrecaent 
between user's and systea's relevance judgaents. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (r«r> rju). If this Z Is sutlstlcally 
significant as Indicated by the asterisks, then resolving anaphors laproves 
the lystea's predications of relevance. 

239 



Page 233 
Appendix E>96 



Page 

A Statistical Co«par1son of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 



AWfERBS 



Correlation Coefficients 



Significance Level 



Q 


s 




184 


1 




-e. 132394 


184 


2 


d 


e. 129099 


184 


1 


• 


d. 064221 


184 


1 




-0. 045491 


184 


1 


J 


0.003093 


184 


1 


M 


0.080914 


184 


1 




-0. 007330 


184 


S 




-0. 148937 


184 


2 


b 


-0. 139155 


184 


2 


c 


-0. 145672 


184 


2 


d 


0. 126770 


184 


2 


• 


0. 142193 


184 


2 


f 


0.039625 


184 


2 


g 


0. 121064 


184 


2 


h 


-0. 032087 


184 


2 


J 


0.036815 


184 


2 


1 


-0.004347 


184 


2 


M 


0. 076227 


184 


2 


n 


-0.024803 



-0. 132581 
0. 129016 
0.063340 
-0.045483 
0.006338 
0. 081955 
-0. 003280 
-0. 149158 
-0. 139231 
-0. 146048 
0. 128531 
0. 140136 
0. 039450 
0. 120839 
-0. 033915 
0. 039314 
-0.004624 
0. 076291 
-0. 023447 



ur 

0.999986 
0.999984 
0.999967 
'9m 99995S 
0- 999797 
0.999973 
0.999852 
0.999987 
0.999996 
0.999990 
0.999986 
0. 999926 
0.999998 
0.999969 
0.999308 
0. 999870 
0. 999980 
0.999966 
0. 999956 



P > .05 



0. 160857 
0.065318 
0. 768379 
-0. 003716 
-0. 719477 
-0.636850 
-1.053561 
0. 195327 
0. 166794 
0. 173707 
0.206309 
0. 764816 
0. 373604 
0. 215494 
0. 219789 
-0. 138526 
0. 195383 
-0. 035013 
-0. 645791 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Slallarlty MeasMrt: #1 » Cosine. #2 • Dice 
TU: Tern Weighting SchcMs: See Result Page R-1 

Correlation coefficients: rjy Is between the user's relevance Judgnent and the 
systOM's predicted relevance based on unresolved anaphors. rjr Is between 
tht user's reltvanct judgmnt and tht system's predicted relevance based 
on rtsolvtd anaphors. 

Btcaust tht ustr's judgMnts were scaltd fnm low to high (1 • most relevant, 
4 • Most non-relevant) a strong ntgatlvt corrtlatlon shows agreoaent 
between user's and systtii's rtltvanct judgMnts. 

Signlflcanct Ltvtl: A posltlvt Z .Indlcatts that tht stcond corrtlatlon Is hightr 
than tht first corrtlatlon (rj,.> rju). If this Z Is statistically 
ffgnlf leant u Indlcattd fey tlSt asttrlsks, thtn rtsolving anaphors linrovts 
tht vsttB's prtdlcatlont of rtltvanct. 
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Appendix E-97 



P«9e 

A Statistical Conparlson of the. Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 



DEFINITE ARTICLE 



Correlation Coefficients Significance Level 



n 


s 






*^ur 


7 


P > 




1 




-0.4243S7 


-0. 422512 


0. 999964 


-I. 198307 




101 


1 


d 


-0. 745425 


-0.735126 


0. 978804 


-0. 360127 




101 




• 


-0.338200 


-0. 354227 


0. 997701 


1.270456 




101 




h 


-0. 084690 


-0. 078579 


0.998527 


-0. 575966 




101 




J 


-0. £05974 


-0. 526366 


0. 947036 


-1.465905 




101 




M 


-0. 742554 


-0.717269 


0. 972555 


-0. 795915 




101 




n 


-0. &92001 


-0. 659707 


0. 968759 


-0. 681743 




101 


2 


• 


-0. 381796 


-0. 379436 


0. 99994X 


-1. 188190 




101 


2 


b 


-0. 328840 


-0. 327643 


0.999988 


-1.289199 




101 


2 


c 


-0. 381283 


-0. 379214 


0.999955 


-1. 165281 




101 


2 


d 


>0. 701048 


-0. 682226 


0. 981310 


-0. 681521 




101 


2 


• 


-0. 339941 


-0.363183 


0.995688 


1. 345865 




101 


2 


f 


>0. 662621 


-0. 658407 


0.974987 


-0. 126532 




101 


2 


5 


>0. 701503 


-0.683738 


0. 960016 


-0. 624396 




101 


2 


h 


>0. 031633 


-0. 030426 


0.999290 


-0. 163398 




101 


2 


J 


-0. 712239 


-0.660565 


0. 978850 


-1. 638745 




101 


2 


1 


-0.001532 


-0. 001474 


0. 999999 


-0. 185714 




101 


2 


m 

• 


-0.731639 


-0. 696171 


0. 981391 


-1.280029 




101 


2 


n 


-0.741866 


-0. 666338 


0. 983399 


-1.964468 





.OS 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: II * Cosine. #2 - Dice 
T»: Teni Weighting SchcMs: See Result Page R-l 

Correlation CotfflclcnU: rju Is between the user's relevance judgnent and the 
systca's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgMnt and the systcn's predicted relevance based 
on resolved anaphors. 

Because the user's Judgnents were scaled fron low to high (1 « Mst relevant, 
4 • Host non-relevant) a strong negative correlation shows agrecncnt 
between user's and systen's relevance JudgtMnts. 

Signlflcanct Level: A positive Z Indicates that the second correlation Is higher 
thin the first corrtUtlon (rjr> rju). If this Z Is statistically 
significant as Indicated by the asterisks, then resolving anaphors Improves 
tht Dftt«i'f prtdlcatlons of relevance. 
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Page 

A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 



DEFINITE ARTICLE 



Correlation Coefficients 



Significance Level 



Qr 


s 


T« r. 






z 












•"ur 










0. 146284 






oX7Sc4 


103 




d 


-O. 053992 




O. 9&A5fil 




1(93 






-^.000526 


-0.000528 


1.000000 


0. 000000 


103 




h 


-0. 319705 


-0. 331746 


0. 987033 


0. 387229 


103 




3 


-0. 316556 


-0.265306 


0.961034 


-0. 937486 


103 




M 


-0. 333513 


-0. 194244 


0.941728 


-2. 050990 


103 




n 


-0. 327129 


-e. 207268 


O. 959997 


-2. 123687 


103 


2 


a 


O. 257744 


0.261709 


0. 999437 


-0. 598744 


103 


2 


b 


0.295230 


0.297466 


0. 999797 


-0. 567829 


103 


S 


c 


0.256569 


0.260124 


0. 999546 


-0. 597584 


103 




d 


-0. 003726 


0.079665 


O. 973030 


-1.764196 


103 


a 


• 


-0. 000526 


-0.000528 


l.OOOOOO 


0. 000000 


103 


2 




0.032856 


0.063885 


0. 979525 


-1.238270 


103 


2 


S 


-0.008358 


0. 072788 


0. 971869 


-1.680514 


103 


2 


h 


-0. 314465 


-0.327134 


0. 994500 


0. 623337 


103 


2 


J 


-0. 350808 


-0.309699 


0. 980181 


-1.064740 


103 


2 


I 


-0.249678 


-0.247160 


0. 998752 


-0. 2S4641 


103 


2 


m 


-0. 213815 


-0. 119610 


0. 965732 


-1.784229 


103 


2 


h 


-0» 242461 


-0. 158955 


0. 972920 


-1.785058 



P > .05 



<*««« 



{»«««■ 
<«««« 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: S1«11ar1t> Measure: #1 > Cdslne. 12 « Dice 
TH: Tcni Weighting Schems: See Result Page R-1 

Correlation Coefficients: rj^ is between the user's relevance judgnent and the 
systea's predicted romance based on unresolved anaphors. nr Is between 
the user's relevance judgMRt and the systen's predicted relevance based 
on rasolved anaphors. 

Because the user's judgaents were scaled fron low to high (1 » most relevant, 
4 > wst non-relevant) a strong negative correlation shows agrecncnt 
between user's and systea's relevance judgaents. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (ri|.> rju). If this Z Is statistically 
significant as Indicated ly the asterisks, then resolving amphors Improves 
the vst«i*s predications of relevance. 



Page 236 
Appendix E>99 



Page 

A Statistical Conparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
- Anaphors and User's Relevance Judgnents: for Anaphoric Class 



DEFINITE ARTICLE 



Correlation Coefficients Significance Level 



Q 


c 






•"ur 


L 


P > 


104 


t 


a 


-0.071262 






W1Q778 




104 




d 


-0.623340 




A QAA7AQ 


— 1. 181 w£9 




104 


1 




-0.429965 


-0. 402667 


0. 980538 


-O. 672547 




104 


1 


h 


-0. 3476S0 


-0. 383169 


0. 987289 


1.059273 




104 


1 


J 


-O. 470712 


-0. 324632 


0.943542 


-2. 035348 


(«»** 


104 


1 




-0.602159 


-0.523198 


0. 955703 


-1. 380366 




104 


1 


n 


-0.415472 


-0.325662 


0. 966071 


-1.618661 




104 




a 


-O. 1539S1 


-O. 154278 


0. 999859 


O. 080141 




104 




b 


-0. 1&1980 


-0. 162270 


0. 999969 


0. 165467 




104 


2 


c 


-0. 149594 


-0. 149793 


0.999884 


O. 059193 




104 




d 


-0.592005 


-0. 479979 


0.948125 


-1.733371 




104 


Z 


• 


-0.426036 


-0. 372469 


0. 9601S5 


-0.919225 




104 




f 


-0.577165 


-0.439319 


0. 938613 


-1.919253 




104 


2 


6 


-0.595353 


-0. 475510 


0. 943964 


-1.78403; 




104 




h 


-0. 178700 


-0.217979 


0.990862 


1.319633 




104 




J 


-0. 472299 


-0.317582 


0. 954676 


-2. 360547 


<*«>»* 


104 


2 


I 


-0. 017455 


-0. 023259 


0. 999859 


1.545265 




104 


2 


M 


-0. 565612 


-0.443978 


0. 956310 


-1.985878 


{»**» 


104 


2 


n 


-0. 384319 


-0. 261436 


0. 968196 


-2. 236528 


<«*** 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 « Cosine. 12 - Dice 
TU: Teni Weighting ScheMs: See Result Page R-1 

Correlation Coefficients: rju is between the user's relevance judgment and the 
systOM's predicted relevance based on unresolved anaphors. rw Is between 
the user's relevance Judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 » most relevant, 
4 • most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Signlflcanct Level: A positive Z Indicates that the second correlation is higher 
than the first correUtion (r4r> rjy). If this Z is statistically 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
tfct lystem't predications of relevance. 
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Appendix E-lOO 
Page 

A Statistical Conparlson of the Relationship BetMoen 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 



DEFINITE ARTICLE 



Correlation Coefficients 



Significance Level 



T« 



ur 



p > . 05 



107 


1 


a 


-0.269170 


-0.269170 


1.000000 


0.000000 


1(97 


1 


d 


-0. 361714 


-0. 435007 


0.964499 


1.215473 


107 


I 


• 


-0.001095 


-0.00109S 


1.000000 


0. 000000 


107 


1 


h 


0. X92S33 


0. 168742 


0.933024 


-0. 185090 


107 


1 


J 


-0.185056 


-0. 254454 


0.975135 


1.308438 


107 


1 


m 


-0. 327307 


-0. 424017 


0.929626 


1. 136070 


107 


1 


n 


-0. 283932 


-0. 416889 


0.902309 


1.313209 


107 


2 


a 


-0. 290729 


-0.290729 




0.000000 


107 


2 


b 


-0.285007 


-0. 265007 


A* WWW 


0.000000 


107 


2 


c 


-0.292260 


-0. 292260 


Am l^PWUnlF 


0.000000 


107 


a 


d 


-0. 356617 


-0. 448451 


0.969943 


1.629910 


107 




• 


-0.001095 


-0.001095 


!• 999999 


9m 909000 


107 


2 


f 


-0. 357481 


-0. 450159 


0.975357 


1.800262 


107 


2 


9 


-0. 356079 


-0. 450427 


0. 970253 


1.679596 


107 


2 


h 


0. 182394 


0.209065 


0. 952498 


-0. 363918 


107 


2 


3 


-0. 089937 


-0. 184122 


0.984029 


2. 183024 


107 


2 


I 


0. 074758 


0.075260 


0. 995797 


-0. 022643 


107 


2 


m 


-0.315778 


-0.430255 


0. 961571 


1.774877 


107 


2 


* 

n 


-0.296199 


-0.411616 


0.968664 


1.957071 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: SiMllarlty Measure: #1 » Cosine. 12 « Dice 
TH: Tera Weighting Schcaes: See Result Page R-1 

Correlation Coefficients: rj^ Is between the user's relevance judgnent and the 
systCM*s predicted re1evar«e based on unresolved anaphors. rjr Is between 
the user's relevance Judgwtnt and the systCM*s predicted relevance based 
on resolved anaphors. 

Because the user's Judgnents were scaled fro* low to high (i • nest relevant, 
4 ■ Most non-relevant) a strong negative correlation shows agrccacnt 
between user's and system's relevance Judgwnts. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rir> rju)* If this Z Is sUtlstlcally 
significant as Indicated bgr tne asterisks, then resolving anaphors Improves 
the iyst«i*s predications of relevance. ^ 
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Appendix E^lOl 
Pege 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 

DEFINITE ARTICLE 



Correlation Coefficients 



Significance Level 



Q 


s 


TM r. 


•"ir 








ill 


J" 


109 


1 


a. 


-0.220519 


-0.220519 


109 


1 


d 


-0.360544 


-0. 394514 


109 


1 




-0.000545 


-0.000545 


109 


1 


h 


-0. 066046 


-0.066349 


109 


1 


J 


-0. 131462 


-0.118022 


109 


1 


m 


-0- 328732 


-0.349701 


109 


1 


rt 


-0.358384 


-0.377749 


109 


a 


* 


-0. 228134 


-0.228134 


109 


£ 


b 


-0. 233510 


-0.233510 


109 




c 


-0. 234593 


-0.234593 


109 




d 


-0. 285177 


-0.305835 


109 


z 




-0.000545 


-0.000545 


109 


z 


f 


-0.293837 


-0. 305177 


109 


z 


o 


-0. 289566 


-0.307594 


109 


z 


h 


-0. 068850 


-0.068934 


109 


z 




-0. 125951 


-0. 126579 


109 


z 


I 


-0. 069214 


-0. 070141 


109 


z 


m 


-0.296651 


-0. 307070 


109 


z 


n 


-0. 330370 


-0. 336207 



ur 

1.000000 

0. 987746 
1.000000 
0.999959 
0.996936 
0.988668 
0.984920 
1.000000 
1.1 



P > .05 



0.987646 
1.000000 
0.988468 
0. 98742S 
0.999958 
0.998624 
0.998880 
0. 987574 
0. 984358 



0.000000 

1.248560 
0.000000 
0. 181646 
-0. 931230 
0. 794758 
0.644409 
0.000000 
0.000000 
0.000000 
0. 739346 
0.000000 
0. 421241 
0. 640S07 
0. 0491 12 
0. 064993 
0. 105703 
0. 373192 
0. 186531 



NOTES : 

Q: Queries 100-199 were seirched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Heasure: #1 » Cosine. «2 » Dice 
TU: Tem Weighting Schemes: See Result Page R*l 

Correlation Coefficients: rjy Is between the user's relevance judgment and the 
system's predicted relevance hised on unresolved anaphors. rjr Is between 
the user's relevance Judgment and the system's predicted relevance based 
on resolved anaphors. 

lecauso the user's Judgments were scaled from low to high (1 • most relevant, 
4 • most non-relevant) a strong negative correlation shows ag rc onent 
betiieiii user's and systesi's relevance Judgments. 

Significance Level: A positive Z indicates that the second correlation Is higher 
them the first corrolatlom {rw> rjy). If this I Is sUtlstlcally 
sigmlf leant as Indicated ly tne asterisks, then resolving amphors Improves 
the ^tfli's predications of relevance. 
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Appendix E>10Z 

Page 

A Ststistical Conparlson of the Relationship Between 
Unresolved Anaphors and Uscr*s Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



DEFINITE ARTICLE 



Correlation Coefficients 



Significance Level 



TU 



Sr 



ur 



P > .05 



135 


1 


135 


1 


135 


r 


135 


1 


135 


1 


135 


1 


135 


1 


135 


2- 


135 




135 


a 


135 


a 


135 




135 


s 


135 




135 


2 


135 


2 


135 


2 


135 


2 


135 


2 



• 

r» -a. 

J 

w -0. 
n -0. 

b 

c -9. 
a -9. 

• -a. 

f -a. 

h -a. 
J -a. 
I -a. 
(It -a. 
n -a. 



.590694 
.79723S 
aai234 
026503 
633972 
796876 
842132 
63763a 
647093 
635240 
818316 
001234 
818960 
816188 
001788 
766710 
001329 
813643 
826860 



-0.600528 

-0.802837 

-0.001234 

-0.054399 

-0.667409 

-0.807115 

-0.819953 

-0.648033 

-0.652253 

-0.644278 

-0.832591 

-0. 001234 

-0.832482 

-0.830510 

-0.002325 

-0. 789706 

-0.001392 

-0.831974 

-0.824281 



0. 999565 
0.992671 
X* 999999 
0-997769 
0. 958150 
0.969432 
0. 987478 
0. 999177 
0.999750 
0.999399 
0.991892 

0. 989473 
0.990944 
0.999999 
0. 977474 

1.000000 

0.988963 
0. 988871 



I. S73369 
0.317165 
0.000000 
1.723191 
0. 623624 
0. 482032 
-0.983305 
1.293648 
1. 184265 
1.309125 
0.791085 
0.O0OO0O 
0.666627 
0. 750372 
1.582846 
0.696317 
0. 479305 
0.859379 
-0. 126964 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Siaflarity Measurt: #1 » Cosine. 12 « Dice 
TV: Tern Weighting SchcMs: See Result Page R-1 

Correlatiofi Coefficients: rjy is between the user's relevance judgnent and the 
syst«i*s predicted relevance based on unresolved anaphors. rj^ is betueen 
tilt ustr'r rtlevaoct Judgwnt and the sjrstca's predicted relevance based 
00 resolved anaphors. 

locauso the user's J u d gtn ts were scaled fro« low to high (1 * aost relevant, 
4 • nst non-relevant) a strong negative correlation shows agrecMnt 
bttwetfi user's and sjfSteM's relevance Judgnents. 

Significanct Level: A posftivt Z indicates that the second correlation is higher 
tbM the first correlation (r4,.> rjy). If this Z is stetistically 
significant as Indlcttod tf urn asterisks, than resolving amphors iiproves 
the ^t«i*s predications of relevance. 
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Appendix E-103 



Page 

A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 



DEFINITE ARTICLE 



Correlation Coefficients 



Significance Level 



TM 



142 
14S 
142 
142 
142 
142 
142 
142 
142 
142 
142 
142 
142 
142 
142 
142 
142 
142 
142 



2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 



d -9. 
m -9. 

h -9. 

J 

M "9, 

r> ^9, 
a -9. 
o -0, 
c -0. 
d -0. 

• -0. 

f 

B -0. 
h -9, 

J 
1 

«i -0. 
•n -0. 



192139 
259997 
000662 
277499 
316609 
304107 
211922 
324310 
439620 
399980 
339496 
000662 
492453 
392275 
277419 
316247 
299325 
359307 
254617 



-0. 189532 
-0. 288940 
-0.000662 
-0. 249826 
-0. 312101 
-0. 303750 
-0. 206682 
'0. 321402 
-0. 434607 
-0. 353568 
-0. 351042 
-0. 000662 
-0. 438539 
-0. 357367 
-0. 054442 
-0. 360107 
-0. 002208 
-0. 356064 
-0. 261328 



ur 

0. 999963 
0. 940953 

1.000000 

0. 914362 
0. 862602 
0. 940922 
0.972119 
0. 999945 
0. 999992 
0.999961 
0.962410 
1.000000 
0. 964745 
0.961686 
0.217178 
0. 848330 
0.398154 
0. 955944 
0. 978172 



P > .05 



-1.337336 
0. 439755 

0.000000 

-0. 302090 
-0. 057208 
-0. 004756 
-0. 098946 
-1.265263 
-1.209647 
-1.251661 

0. 1964S2 

0. 000000 
-0. 255435 

0. 085668 
-0. 804482 

0. 370S05 
-1.037952 
-0. 051074 

0. 140630 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: SlMllarlty Measure: #1 - Cosine. 12 - Dice 
TH: Teni Weighting ScheMs: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance Judgment and the 
systea's predicted relevance based on unresolved anaphors. rjr Is between 
the user*s relevance judgaent and the system's predicted relevance based 
on rttolved anaphors. 

lecaute the user's judgMnts were scaled froai low to high (1 » most relevant* 
4 • Mft non-relevant) a strona negative correlation shows agreenent 
between user's and syste«'t relevance judgnents. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rir> rju). If this Z Is statistically 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the DfftaB's predications of relevance. 
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Page 

A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



DEFINITE ARTICLE 



Correlation Coefficients Significance Level 

^ ^ ^iu ""jr %r 2 > 



158 

ISB 

ISA 

158 

158 

158 

158 

153 

158 

158 

158 

158 

158 

158 

158 

158 

158 

158 

158 



1 
1 
1 
1 
1 
1 
1 

a 

2 
2 
S 
2 
2 
2 
2 
2 
2 
2 



• 9 

d -0 

• -0. 

h -0. 

J 

M -0. 

r. -0. 

• 0. 
b 0. 

C 0. 

d -0. 

• -0. 

f 0. 

B -0. 

h -0. 

J -0. 

1 -0. 

,iw -0. 

Vi -0, 



. 038318 
. 1&S911 
001032 
261133 
172847 
212637 
190370 
090669 
174475 
116130 
113967 
001032 
016314 
039306 
136551 
103790 
007262 
131206 
X20778 



0. 037274 
-0. 142162 
-0. 001032 
-0. 249736 
-0. 138819 
-0. 184153 
-0. 171699 
0. 089328 
0. 173910 
0. 115103 
-0. 096088 
-0.001032 
0. 025684 
-0. 073014 
-0. 141136 
-0. 098236 
-0.010593 
-0. 1 17746 
-0. 118780 



0. 999861 


0. 234405 


0. 977468 


-0. 370037 


1.000000 


0. 000000 


0. 993442 


-0. 384794 


0.969187 


-0.519199 


e. 975315 


-0. 489168 


0.972976 


-0. 305557 


0. 999809 


0. 257478 


0. 999972 


0. 285531 


0.99987S 


0. 244216 


0. 978496 


-0. 324375 


1^000000 


0. 000000 


0. 981859 


-0.184111 


0. 977661 


-0. 289364 


0. 993226 


0. 148835 


0. 987446 


-0. 131824 


0. 999166 


0. 305304 


0. 983213 


-0. 277017 


0. 981990 


-0. 039666 



NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: SiMllarlty Measure: #1 • Cosine. 12 • Dice 
TU: Tern Weighting Scheaes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance Judgment and the 
systeiR's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgnent and the systaa's predicted relevance based 
on resolved anaphors. 

Because the user's judgnents were scaled from low to high (1 • nost relevant, 
4 > Rost non-relevant) a strong negative correlation shows agreonent 
between user's and systen's relevance judgMnts. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rjr> fju). If this Z Is sutlstlcally 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the lystM's predications of relevance. Q>f o 
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Appendix E>105 
P«9e 

A Statlstlcil Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



DEFINITE ARTICLE 



Correlation Coefficients 



A 
W 




TW r. 














179 


1 




-0.716613 


-0. 7:6201 


170 


1 


d 


-0. 731051 


-0. 740470 


I7ld 


1 


• 


-0.001103 


-0.001109 


170 


1 


h 


0. 166612 


0. 161113 


170 


1 


J 


-0. 594562 


-0. 634013 


170 


1 


M 


-0. 636259 


-0. 653973 


170 


1 


n 


-0. 515708 


-0. 550422 


170 


2 


a 


-0. 678385 


-0. 6781 12 


170 


2 


b 


-0. 637558 


-0. 637414 


170 


2 


c 


-0. 678090 


-0. 677830 


170 


2 


d 


-0.691708 


-0. 716829 


170 


2 


• 


-0.001109 


-0.001109 


170 


2 


f 


-0. 696939 


-0. 729204 


170 


2 


9 


-0. 689511 


-0.717631 


170 


2 


h 


0. 153761 


0. 148671 


170 


2 


J 


-0. 553947 


-0. 594241 


170 


2 


1 


0. 045678 


0. 044916 


170 


2 


P» 


-0. 623943 


-0. 655047 


170 


2 


n 


-0. 533728 


-0. 577632 



Significance Level 



r 

ur 


z 


P > 


0. 999987 


-e. 500158 




0. 993749 


0. 536490 




1.000000 


0. 000000 




0. 999923 


1.938776 




0.987689 


1.318577 




0. 992766 


0. 822547 




0. 992263 


1.377149 




0. 999979 


-0. 249519 




0.999996 


-0. 27450S 




0. 999983 


-0. 262757 




e. 995616 


1.497348 




1.000000 


0. 000000 




e. 995410 


1.803287 




0. 995677 


1.648623 




0. 999951 


2. 235455 


<«««« 


0. 994575 


1.866306 




0. 999997 


1.372835 




0. 995500 


1.682713 




0. 994568 


1.967606 


<«»»» 



NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: II « Cosine. 12 « Dice 
TV: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rw Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's Judgments were scaled from low to high (1 * most relevant. 
4 ■ most non-relevant) a strong negative correlation shows agreanent 
between user's and system's relevance Judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rjr> rju). If this Z Is statistically 
significant as Indicated by the asterisks, then resolving anaphors Improves 
the iytt«i'i predlcatloni of relevanct. 
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Page 

A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judg«ents with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 



DEFINITE ARTICLE 



Correlation Coefficients Significance Level 



Q 


s 


™ '}u 




Ur 


z 


• OA 

18V 


1 


a 


-0. 297538 


—0. S99409 


0. 999723 


0. 416233 




1 


d 


-0. 475084 


-0. 465962 


0. 972956 


-0. 222534 


180 


I 


• 


-0.484854 


-0. 448249 


0. 943001 


-0. 609328 


180 


1 


h 


-0. 234297 


-0. 273204 


0. 994010 


1.815255 


180 


1 


J 


-0. 395090 


-0. 383178 


0. 972203 


-0. 274375 


180 


1 


m 


-0. 473372 


-0. 462677 


0. 970381 


-0. 248950 


180 


1 


rt 


-0. 428501 


-0.421864 


0.971957 


-0. 155029 


180 


a 


• 


-0. 219548 


-0. 222590 


0. 999568 


0. 530523 


180 


2 


b 


-0. 157482 


-0. 159036 


0. 999904 


0. 573887 


180 


2 


c 


-0.213261 


-0. 21 5989 


0. 999684 


0. 554789 


180 


& 


d 


-0. 475058 


-0. 465784 


0. 965917 


-0. 201648 


180 


2 


• 


-0. 434632 


-0. 423526 


0. 92S996 


-0. 160429 


180 


2 


f 


-0. 393963 


-0. 397721 


0. 966582 


0. 079270 


180 


2 


9 


-0. 468597 


-0. 462149 


0. 965305 


-0. 136573 


180 


2 


h 


-0. 352302 


-0. 400961 


0. 981 130 


1.333672 


180 


2 


J 


-0. 449142 


-0. 425860 


0. 976790 


-0. 599360 


180 


2 


1 


-0. 126480 


-0. 150684 


0. 997611 


1.761979 


180 


2 


m 


-0. 476596 


-0. 459917 


0. 969177 


-0. 380275 


180 


2 


• 


-0. 445123 


-0. 424273 


0. 974688 


-0. 513802 



NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: similarity Measure: 11 « Cosine. #2 • Dice 
TH: TerM Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the systeN's predicted relevance based 
on resolved anaphors. 

lecause the user's judgments were scaled from low to high (1 • most relevant, 
4 • most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rir> rju). If this Z Is statistically 
significant as Indicated by the asterisks, then resolving anaphors Improves 
the lyitem's predications of relevance. 250 
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Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



DEFIWITE ARTICLE 



Correlation Coefficients 



Significance Level 



Q 


s 


TW r. 




r 


2 












ur 




182 


1 




-e. 194413 


-0. 192936 


0. 999984 


-1. 366175 


lez 


1 


d 


221779 


-0. 187207 


0. 993105 


-i . 563342 


162 


1 


• 


-0. 000462 


-0. 000462 


1.000000 


0. 000000 


182 


1 


h 


0. 165728 


0.201367 


0. 997233 


-1. 131017 


182 


i 


J 


-0. 006062 


0. 002560 


0. 992515 


-0. 45936S 


182 


1 


m 


-0. 16736S 


-0. 154416 


0. 96840S 


-1. 160061 


182 


1 


n 


-0. 166005 


-0. 135654 


0. 982471 


-0. 667261 


182 


2 


* 


-0. 174304 


-0. 172579 


0. 999979 


-1.414123 


182 


2 


b 


-0. 101067 


-0. 100631 


0. 999999 


-1.424118 


182 


2 


c 


-0. 137425 


-0. 136297 


0. 999991 


-1.417226 


182 


2 


d 


-0. 223362 


-0. 177736 


0. 991122 


-1.637506 


182 


2 


• 


-0. 037327 


-0. 037327 


1.000000 


0. 000000 


182 


2 


f 


-0. 147249 


-0. 126453 


0. 993692 


-0. 986232 


182 


2 


B 


-0. 195747 


>0. 158885 


0. 991&35 


-1.527192 


182 


2 


h 


0. 188673 


0. 198303 


0. 999043 


-1. 163623 


:62 


2 




-0. 009634 


0. 012329 


0. 987399 


-0. 736875 


182 


2 


I 


0. 157129 


0. 162794 


0. 999685 


-1.206232 


182 


2 


.m 


-0. 198272 


-0. 152594 


0. 987349 


-1.538783 


182 


2 


n 


~0. 190478 


-0. 151898 


0. 984652 


-1. 160670 



P > .05 



NOTES: 

0: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: II « Cosine. 12 - Dice 
TW: Term Weighting Schemes: Sec Result Page R-1 

Correlation Cotfffclents: rju Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved amphort. 

Because the user's judgments were scaled from low to high (1 « most relevant, 
4 > most non-relevant) a strong negative correlation shows agreonent 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rjr> rjy). If this Z Is statistically 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the liystem's predications of relevance. ^ 
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Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



DEFINITE ARTICLE 



Significance Leve l 



Z p > .05 



0. I60SS7 

1.010003 
-0.511292 
0. 224231 
0. 600278 
0. 713418 
0. 195327 
0. 168794 
0. 173707 
1.052771 
0. 927009 
0. 915649 
1.071616 
0. 322612 
0. 786557 
-0.017192 
0. 878859 
0. 821506 



NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: II - Cosine. #2 - Dice 
TW: Term Weighting Schemes: See Result Page R-l 

Correlation Coefficients: rju Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

lecause the user's judgments were scaled from low to high (1 « most relevant, 
4 ■ most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rjr> rjy). If this Z Is sutlstlcally 
significant as Indicated by the asterisks, then resolving anaphors Improves 
the Dfstem's predications of relevance. . 



Correlation Coefficients 



TW 



ju 



jr 



ur 



184 

184 

184 

184 

184 

184 

184 

184 

184 

184 

184 

184 

184 

184 

184 

184 

184 

184 

184 



1 
1 
1 

1 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 



« -0. 

d 0, 

• 0. 

h -0. 

J ». 
m 0. 
n -0. 

« -0. 

b -0. 

c -0. 

d 0. 
V 0. 

f 0. 

g 0. 

iy -0. 

J 0. 

1 -0. 

in 0. 

n -0. 



. 132394 
. 129099 
. 064221 
. 045491 
. 003093 
080914 
007330 
148937 
1391 55 
145872 
126770 
142193 
039625 
121064 
032087 

038815 
004347 
076227 
024803 



-0. 132581 
0. 066967 
0. 004749 
-0.042961 
-0. 010500 
0. 039387 
-0. 045538 
-0. 149158 
-0. 139231 
-0. 146048 
0. 054595 
0. 082934 
-0. 025722 
0.044100 
-0. 032241 

-0. 012^46 
-0. 004342 
0.015787 
-0. 069690 



0. 999986 
0. 964340 
0. 965202 
0. 999755 
0. 963262 
0. 962558 
0. 971267 
0. 999987 
0. 999998 
0. 999990 
0. 949794 
0. 958561 
0. 948904 
0. 947886 
0. 999998 
0. 957412 
0. 999999 
0. 952486 
0. 970039 
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A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



CENTRAL PRONOUNS 
Correlation Coefficients Significance Level 



Q 


s 




J" 


ur 


Z 


S03 




a 


-0. 10S343 


-0. 106148 


0. 999980 


0. 553864 


di93 




d 


-0. 141626 


-0. 146761 


0. 988414 


0. 148595 


203 




• 


-0.001022 


-0.001022 


1.000000 


0.000000 


203 




h 


0. 1 17064 


0. 119714 


0. 995724 


-0. 125785 


203 




3 


0. 072857 


0. 069456 


0. 988791 


0. 099277 


203 




m 


-0. 097276 


-0. 103047 


0.988140 


0. 164159 


203 




rt 


-0.012827 


-0. 014390 


0.994124 


0. 062847 


203 


2 


m 


-0. 042574 


-0. 043710 


0. 999964 


0.582901 


203 


2 


b 


0. 005796 


0. 005370 


0.999995 


0. 574551 


203 


2 


c 


-0. 037613 


-0. 038579 


0. 999974 


0. 585132 


203 


2 


d 


-0. 070321 


-0. 053172 


0. 986919 


-0. 463065 


203 


2 


• 


-0.001022 


-0.001022 


1.000000 


09 009000 


203 


2 


f 


-0. 036263 


-0. 021549 


0. 991421 


-0. 469875 


203 


2 


9 


-0. 070989 


-0. 053954 


0. 987333 


-0. 467457 


203 


2 


h 


0. 069005 


0. 074765 


0. 998039 


-0. 401902 


203 


2 


J 


0. 034705 


0. 046557 


0. 986288 


-0.312227 


203 


2 


1 


0. 002329 


0. 002661 


0. 999996 


-0. 498452 


203 


2 


M 


-0. 044466 


-0. 029961 


0. 985690 


-0. 374055 


203 


2 


• n 


0. 004985 


0.015115 


0. 992221 


-0. 354036 



P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychtNPO 
S; Similarity Measure: #1 * Cosine. #2 « Dice 
TH: Term Weighting Schemes: See Result Page R.l 

Correlation Coefficients: rjy Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rir Is between 
the user's rtlevanct judgment and the system's predicted relevance based 
on rtsolvtd anaphors. 

Because the user's judgments were scaled from low to high (1 • most relevant. 
4 • Bost non-reltvant) a strong negative correlation shows agreoiient 
between user's and system's relevance judgments. 

Slgnlflcanct Level: A positive Z Indicates that the second correlation Is higher 
thin the first correlation (rjp> rj„). If this Z Is sUtlstlcally 
slgnlflcint as Indicated by the asterisks* then resolving anaphors Improves 
the vttea's prodlcatlons of relevance. 
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Appendix E-110 
Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgments: for AnaphJrIc CUss 

CENTRAL PRONOUNS 



Correlation Coefficient* 



Significance Level 



TW 



£07 




a 


0. 069726 


207 




d 


-0. 006064 


S07 




• 


-0. 000956 


207 




h 


0. 213822 


£07 




J 


0. 168212 


207 




M 


0.099118 


207 




n 


0. 0646S5 


207 


2 


» 


0.969602 


207 


2 


b 


0. 0S4739 


207 


2 


c 


0. 070263 


207 


2 


d 


0. 021932 


207 


2 


• 


-0.0009S6 


207 


2 


f 


-0. 040281 


£07 


2 


0 


0. 018805 


207 


2 


h 


0. 202720 


207 


2 


J 


0. 193024 


207 


2 


1 


0. 175987 


207 


2 


m 


0. 121846 


207 


2 


'n 


0. 096820 



0. 088733 
0.008881 

-0.000956 
0. 204871 
0. 150318 
0. 097251 
0. 072536 
0. 068528 
0. 054262 
0. 069245 
0. 053102 

-0. 000956 
0. 031567 
0. 058703 
0. 194292 
0. 1S9S61 
0. 172265 
0. 138912 
0. 122376 



ur 

0. 999863 
0. 959878 
1.000000 
0.999818 
0. 9971 12 
0. 980516 
0. 979105 
0.999717 
0.999936 
0. 999769 
0.967664 
1.000000 
0. 953723 
0. 966936 
0. 999825 
0. 999021 
0. 999885 
0. 985794 
0. 964216 



P > .05 



0. 269908 
-0. 235964 
0. 000000 
2. 1 16904 
1.064410 
0.042517 
-0. 168435 
0. 202447 
0. 188977 
0. 212202 
-0. 548712 

0. 000000 

-1.058178 
-0. 694770 
2. 029956 
0. 356631 
1. iX3£86 
~0. 45664£ 
-0. 596537 



<•««« 



<«»»« 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: II ■ Cosine. #2 • Dice 
TH: Term Weighting Schemes: See Result Page R-1 

^"'*I!2III.?I2J2S?' ,13" ^* J***^ '^•'•vance Judgment and the 

system s predicted relevance based on unresolved anaphors. rw 1$ between 

STrllSl L '"^ predicted relevance based 



??«!**Il."!irI* ^Jf'^!* to high (1 . most relevant, 

jrmost non-rtlevant) a strono negative correlation shows agreoient 
between user's and system's relevance Judywnts. 
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Appendix E-lll 
Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



CENTRAL PRONOUNS 



Correlation Coefficients 



Significance level 



ur 



P > .05 



S12 




« 


-0. 532393 


-0. 534533 


0. 999965 


1.423618 






d 


-0. 630979 


-0. 642554 


0. 994251 


0. 678550 


S12 




• 


-0.2aifi44 


-0.387175 


0. 966010 


2. 121565 


&i& 




h 


0. 089775 


0. 084396 


0.988412 


0. 173772 


&1Z 




J 


-0. 592539 


-0. 578060 


0. 978164 


-0.418296 


ZiZ 




m 


-0. 663788 


-0. 651 190 


0. 994946 


-0.801316 


&1Z 




n 


-0. 701^85 


-0. 653017 


0. 986907 


-2.929199 


£l& 


2 


a 


-0. 553674 


-0. 556193 


0. 999956 


1.510422 


£i£ 


2 


b 


-0. 5361 18 


-0. 537247 


0. 999990 


1.407011 


£i£ 


2 


c 


-0. 560078 


-0. 562278 


0. '^99967 


1.525915 


222 


2 


c 


-0. 683127 


-0. 628426 


0. 947118 


-1.070159 


212 


2 


• 


-0.288411 


-0. 387069 


0.975143 


2. 230195 


212 


2 




-0. 681729 


-0. 602285 


0. 937197 


-1.380904 


212 


2 


g 


-0. 692679 


-0. 630822 


0. 943921 


-1. 176219 


212 


2 


n 


0. 055268 


0. 047928 


0.992705 


0. 279549 


212 


2 


J 


-0. 685346 


-0.606738 


0. 942597 


-1.427929 


212 


2 




0. 004343 


0. 003638 


0. 999915 


0.264216 


212 


2 


m 


-0.710908 


-0. 637601 


0. 941238 


-1.363292 


212 


2 


• ri 


-0.731150 


-0. 632344 


0. 935889 


-1.717563 



<««*« 



<»»«» 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 « Cosine. 12 » Dice 
TU: Term Weighting Schemes: See Result Page R-l 

Correlation Coefficients: rju 1$ between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rw Is between 
the user*s relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the u%€r*$ judgments were scaled from low to high (I • most relevant, 
4 • most non-relevant) « strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
thill the first correlation (r4,.> rjy). If this Z Is statistically 
significant as Indicated by the asterisks, then resolving anaphors linroves 
tht ^st«m*s predications of relevance. 
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Page 

A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 



CENTRAL PRONOUKS 
Correlation Coefficients 



Significance Level 



Q 


s 


oe A 




Z13 




219 




219 




219 




219 




219 




219 


2 


219 


2 


219 


2 


219 


2 


219 


2 


219 


2 


219 


2 


219 


2 


219 


2 


219 


2 


2.19 


2 


219 


2 



TU 



a -e. 331415 
d -0.431196 

• -0.001101 

h -0. 738016 
J -0.671656 
m -0.541866 
n -0.562968 

• -0.253123 
b -0.223322 
c -0.264359 
d -0.395491 

• -0.001101 
f -0.380616 
g -0.404803 
h -0.769917 
J -0.594641 
1 -0.616435 
m -0.504693 

•n -0.520418 



-0. 331415 
-0. 461884 

-0. 001101 

-0. 743589 

-0-711695 
-0. 580514 
-0. 600416 
-0. 253123 
-0- 223322 
-0. 264359 
-0. 427720 
-0. 001 101 
-0.413374 
-0. 437486 
-0. 747135 
-0. 640516 
-0. 650398 
-0. 549308 
-0. 562119 





z 


ur 




1. 000000 


0. 000000 


0. 984751 


0. 777714 


1.000000 


0. 000000 


0. 972328 


0. 142141 


0. 9824S3 


1. 120865 


0. 983723 


1.006440 


0. 981076 


0. 923600 


1.000000 


0. 000000 


1.000000 


0. 000000 


1.000000 


0. 000000 


0. 981014 


0. 720933 


1.000000 


0. 000000 


0. 975909 


0. 647394 


0. 980249 


0. 72004A 


0. 963845 


-0.519063 


0. 981724 


1. 164429 


0. 977838 


0. 816354 


0. 982342 


1 . 085S09 


0. 979712 


0.962351 



P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Slallarlty Measure: II « Cosine. #2 • Dice 
TU: Tcra Weighting Schcaes: See Result Page R-1 *' 

Correlation Coefficients: rjy Is between the user's relevance Judgaent and the 
systiH*s predicted relevance based on unresolved anaphors. rw Is between 
the user's relevance judgMent and the system's predicted relevance based 
on resolved amphors. 

lecause tlie user's judgMnts were scaled fron low to high (1 « aost relevant, 
4 • aost non-relevant) a strong negative correlation shows agreoMnt 
between user's and tystea't relevance judgwnts. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
tfcan the first correlation (n^y rj„). If this Z Is sutlstlcally 
sffRlf leant as Indicated ty the asterisks, then resolving anaphors Iwroves 
tfct 9jf»tm^% predications of relevance. 256 
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Page 

ii f Statistical Coiiparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgments: for AnaphSric CU« 



CENTRAL PRONOUNS 



Correlation Coeff1ci#nt< 



Significance Level 



TM 



2S1 




a 


0. 044923 


221 




d 


0. 075409 


221 




• 


-0.000889 


221 




h 


-0.034912 


221 




J 


-0. 072S48 


221 




M 


0.012463 


222 




n 


-0. 082838 


221 


2 


a 


0. 03S969 


221 


2 


b 


0.024068 


221 


2 


c 


0. 035314 


221 


2 


d 


0.040988 


221 


2 


• 


-0.000885 


221 


2 


f 


-0. 003799 


221 


2 


9 


0.035177 


221 


2 


h 


-0. 034210 


221 


2 


J 


-0. 066140 


221 


2 


I 


-0. 001531 


221 


2 


M 


-0.013498 


221 


2 


* n 


-0. 092854 



0. 044756 
-0. 091837 
-0. 000885 
-O. 037803 
-O. 179923 
-0. 140895 
-O. 191504 
0. 035454 
0. 023727 
0.035149 
-0.090963 
-0.000885 
-O. 088283 
-O. 091772 
-O. 036675 
-O. 147974 
-0. 001678 
-O. 1 14270 
-O. 156450 





2 


ur 




0. 999961 


0. a»7ll99 


0.953991 


2. 082487 


1.000000 


0. 000000 


0.999912 


0. 614601 


0. 980257 


2. 031937 


0. 967212 


2. 257995 


0. 987330 


2. 554274 


0. 999933 


0. 037527 


0.999973 


0. 173684 


0. 999942 


0. 057249 


0.956845 


1.690999 


1.000000 


0. 000000 


0.971130 


1.319809 


0. 957515 


1.639048 


0. 999963 


1.076885 


0. 983964 


1. 717527 


1. 000000 


399711 


0. 975392 


1.707021 


0. 989795 


1.673295 



P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: SlalUrlty Measure: #1 • Cosine. #2 • Dice 
TM: Tera Weighting Schcoes: See Result Page R-1 ** 

^"'•Ilo!Il.?H2fiS?' relevance Judgment and the 

system s predicted relefance based on unresolved anaphors. tir is between 

i^JISr L'*'*!!?* the system's predicted relevance based 

on rtsoivto •mphors* 

J^lli!lf*lll.'***r** ^ • «»t relevant. 

LlJSl . "•^•'•^•"t) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 
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A SUtlstlcil Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 



CEHTRAL PRONOUNS 
Correlation Coefficients 



Significance Level 
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222 




222 
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222 


2 


222 
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222 
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222 
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222 
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222 
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222 
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222 
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222 
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222 


2 


222 
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TU 

a -0. 

d -e. 

• -0. 

h -0. 

J -0. 
m 

n -0. 

• -0. 

b -0. 
e -0. 
d -0. 

• -0. 
f -0. 
g -0. 
h -0. 

J -0. 
I -0. 
w -O, 
*n -0. 



. 065303 
.407137 
.000884 
. 182421 

403934 
430564 
194080 
.076259 
.063133 
. 073564 
.410846 
» 00000^ 
.338840 
410581 
022308 

430670 
002454 
442443 
256865 



-0. 065303 
-0. 374690 
-0.000884 
-0. 182934 

-0. 376705 
-0. 399497 
-0. 160212 
-0. 076259 
-0.063133 
-0. 073564 
-0.391805 
-0.000884 
-0. 336722 
-0. 393432 
-0.021811 

-0. 407286 
-0. 002434 
-0. 421587 
-0. 223773 



ttr 

1.000000 
0. 948363 

1.000000 

0.999813 

0.938304 

0. 942107 

0. 975691 

1.000000 

1.000000 

1.000000 

0. 980736 

1.000000 

0. 994392 

0.987102 

0.999995 

0. 972594 
1.000000 
0. 979479 
0. 984624 



P > .05 



0. 000000 

-0. 490593 
0. 000000 
0. 120842 

-0. 377074 
-0. 449155 
-0. 697601 
0. 000000 
0.000000 
0.000000 
-0. 472869 
0.000000 
-0. 095047 
-0. 5201 66 
-0. 702709 

-0. 491216 
-0. 223601 
-0. 509322 
-0.867158 



NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Siailarity Measure: #1 » Cosine. #2 » Dice 
TU: Tern Weighting Schcaes: See Result Page R-1 

Correlation Coefficients: rjy is between the user's relevance judgment and the 
systea's predicted relevance based on unresolved anaphors. rim is between 
the user's rtltvance judgMnt and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgMnts were scaled fro« low to high (1 « Mst relevant, 
4 > Most non-relevant) a strong negative correlation shows agrecnent 
between user's and system's relevance Judgwnts. 

Significance Level: A positive Z indicates that the second correlation is higher 
than the first correlation (rj^> rjy). If this Z is sUtistically 
significant as Indluted by the asterisks, then resolving anaphors inproves 
the Kyst«i's predications of relevance. 258 
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A Statistical Conparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judg.ents: fo? Anaph^Hc ClHs 

CENTRAL PRONOUNS 



Correlation Coefficients 



Significance Level 



Q 


s 




1 


SS3 
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1 


223 


1 


223 


1 


223 


i 


223 


1 


223 


2 


223 


2 


223 


2 


223 


2 


223 


2 


223 


2 


223 


2 


223 


2 


223 


2 


223 


2 


223 


2 


222 


2 



TW 



h 
J 

M 

n 
m 

b 
c 
d 



9 

h 

J 
1 
m 
n 



-9, 732594 
694873 
-0. 000648 
-0. 373036 
-0. 422509 
~0. 695676 
-0. 639829 
-0. 754136 
-0. 718551 
-0. 752835 
-0. 732325 
-9. 000648 
-0. 709220 
-0. 731842 
-0. 361158 
-0. 389875 
-0.315603 
-0. 726692 
-0. 670259 



-0. 732594 
-0. 677763 
-0. 000648 
-O. 373031 
-0.420913 
-0. 686277 
-0. 629727 
-0. 754136 
-O. 718551 
-0. 752835 
-0. 707707 
-0.000648 
-O. 699728 
-0. 706009 
-0. 361037 
-0. 391240 
-0. 315725 
-0. 732142 
-0. 678302 



ur 

1. 000000 
O. 997143 
1.000000 
1.000000 
O. 999851 
O. 997616 
O. 997056 
1. 000000 
1.000000 
1.000000 
0.986766 
1.000000 
O. 971249 
0. 983403 
1.000000 
0. 999863 
1.000000 
0. 988856 
O. 989974 



P > .05 



O. 000000 
-1.092917 
0. 000000 
-O. 138144 
-O. 380351 
-O. 687806 
-O. 626973 
0.000000 
0.000000 
0.000000 
-O. 790231 
O. 000000 
-O. 209693 
-0. 743238 
-O. 982439 
O. 334508 

-I. 159576 
0. 199866 
0. 267234 



NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Sinilarity Measure: #1 » Cosine. #2 • pice 
T«: Ter* Weighting Schcnes: See Result Page R-l 

Correlation Coefficients: rjy is between the user's relevance Judgment and the 
systen s predicted romance btsed on unresolved anaphors. rtr is between 
the user's relevance judgnent and the systcn's predicted relevance based 
on resolved aniphors. 

Because the user's Judgwnts were scaled froo low to high (l • nost relevant, 
4 • m$t non>relevant) a strong negative correlation shows agreonent 
Ntwetn user's and systeii's relevance judgnents. 

Significajcj level; A positive Z indicates that the second correlation is higher 
thin the first correlation (rj,.> riy). If this Z is statistically 

J2 .lifllf " *I!?*^f*^ tytRe asterisks, then resolving anaphors improves 
tm ^stem's predications of relevance. 
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•I f Statistical Coaparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with ResoWm 
Anaphors and User>s Relevance Judgments: for AnaphJrIc Cull 



CENTRAL PRONOUNS 



Correlation Coefficient* 



Significance Level 



TH 



227 
227 
227 
227 

227 

227 

227 

227 

227 

227 

£27 

227 

227 

227 

227 

227 

227 

227 
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1 
1 
1 

1 
1 
1 
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2 
2 
2 
2 
2 
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2 
2 
2 
2 



J 
ta 
n 
m 
b 
e 
d 



9 

h 

J 
1 
m 
.n 



> -e. 128573 
I -0.321199 
-0.0M65a 
•. 333399 

059223 
••0.227A12 
-0.061523 
-0. 122705 
-0. 133699 
-0. 126353 
-0.286205 
-0.000658 
-0. 282278 
-0. 290104 
0. 331148 
0. 023736 
0.037912 
-0.214012 
-0. 066552 



-0. 128973 
-0. 366376 
-0.000658 
0.338388 
-0.078257 
-0.318183 
-0. 200120 
-0. 122705 
-0. 133699 
-0. 126353 
-0. 300352 
-0. 000658 
-0.301705 
-0. 302799 
0. 335086 
-0. 053345 
0. 085841 
-0. 252270 
-0. 146941 





z 






1 . 000000 


V. WWW 


0. 951639 


0. 690586 


4 • www 


0. 000000 


0. 990355 


-0. 170894 


0. 942847 


1.830749 


0. 947080 


1.284844 


0. 913762 


1.512095 


1.000000 


0.000000 


1.000000 


0.000000 


1.000000 


o.oooooo 


0. 955781 


0.191304 


l.OOOOOO 


0. 000000 


0. 952469 


0. 294832 


0. 9S2048 


0. 192151 


0. 998555 


-0. 347107 


0. 976766 


1.602716 


0. 989605 


-1.489756 


0. 96646S 


0. 678814 


0.962811 


1.327089 



P > .05 



NOTES: 

0: Queries lOO-igg were searched on INSPEC: 200-299 on PsychlNFO 
S: SiMllarlty Neasurt: #1 • Cosine. #2 • Dice 
TH: Terai Weighting SchMnet: See Result Page R-l 

^""•liJllI.?*^? f ?' JA» relevance Judgmtnt and the 

JC**i*.'"'**J**^ remanct based on unresolved anaphors. r^ls between 
llTrllSwLlliSSJJ.'^"'^* '"^ predated releJance bisS 

K'HIf***** *"*r*' »caled froii low to high (1 • most relevant, 

LLS*..!;:::*'*!*"*^* ^''^elatlon shJws agre«pent 

between user's and sys ten's relevance judgments. 

SlgnlflMjct Level: A positive Z Indicates that the second correlation Is higher 
.5II«JJ* ^l^^ wrrtlatlofl (r«^> rjy). If this Z Is statistically ' 
liTlVA^J^} " Jy.tfie asterisks, then resolving anaphors Improves 



tht lyttiM's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



CENTRAL PRONOUNS 



Correlation Coefficients 



Significance Level 



n 

M 


c 

9 


TW r. 






z 












''ur 




230 


1 


• 


r-0. 373570 


-0. 373570 


1.000000 


0. 000000 


£30 


1 


d 


-0. £51948 


-0. £30046 


0. 986598 


-0. 599926 


&39 


1 


• 


-0.000600 


-0. 000600 


1.000000 


0. 000000 


£30 


1 


h 


0. 076769 


0. 07a4£9 


0. 999973 


-0. 981999 


£30 


1 


J 


0. 058008 


0. 063166 


0. 998£66 


-0.382513 


£30 


1 


m 


-0. 10££39 


-0. 079651 


0. 9689£1 


-0. 396626 


£30 


1 


n 


-0. 169797 


-0. 1£67£6 


0. 970358 


-0. 779544 


£30 


£ 


a 


-0. 3056£0 


-0. 3056£0 


1.000000 


0. 000000 


£30 


£ 


b 


-0. £34657 


-0. £34657 


1.000000 


0. 000000 


£30 


£ 


c 


-0. 301 £74 


-0. 301 £74 


1. 000000 


0. 000000 


£30 


£ 


d 


-0. 19753£ 


-0. 19£606 


0. 998371 


-0. 367888 


£30 


£ 


• 


-0. 000600 


-0. 000600 


1.000000 


0. 000000 


£30 


£ 


f 


-0. 149564 


-0. 148999 


0. 999690 


-0. 099939 


£30 


£ 


B 


-0. 196159 


-0. 191 49£ 


0. 998626 


-0. 395459 


£30 


£ 


h 


0. 064555 


0. 065137 


0. 999985 


-0. 465699 


£30 


£ 


J 


0. 074391 


0. 07£703 


0. 999S£1 


0. 200236 


£30 


£ 


1 


0. 073664 


0.074166 


0. 999994 


-0. 609238 


£30 


£ 


M 


-0.07513£ 


-0. 070767 


0. 996823 


-0. 239286 


£30 


£ 


. n 


-0. 137167 


-0. 1£5737 


0. 997244 


-0. 676492 



P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: SImlUrlty Neasurt: #1 • Cosine. 12 • Dice 
TW: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance judgment and the 
system*! predicted relevance based on unresolved anaphors. rjr Is between 
the ustr^s rtltvance judgment and the system's predicted relevance based 
on rtsolved anaphors. 

Because tJie user's judgments were scaled from low to high (1 « most relevant, 
4 • most non-relevant) a strong negative correlation shows agreement 
Ntween user's and system's relevance judgments. 

Signlflcanct Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (ri^> rju). If this Z Is statistically 
significant as Indicated by the asterisks, then resolving anaphors Improves 
tilt lyftM's predications of relevance. ' ^ 
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Page 

J Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



CENTRAL PRONOUNS 



Correlation Coefficients 



Significance Level 



Til 



£35 
£35 
£35 
£35 

£35 

£35 

£35 

£35 

£35 

£35 

£35 

£35 

£35 

£35 

£35 

£35 

£35 

£39 
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£ 
£ 
£ 
£ 
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£ 
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• -0.«311£3 
d -0.3£9375 

• -0. 000931 
h -0.094045 



J --•0. 

w -0. 

n -0. 

m -0. 

b -0. 

c -0. 

d -0. 
• -0. 

f -0. 
g -0. 
h -0. 

J -0. 

1 -0. 

m -0. 

.r» -0, 



506407 
397157 
457890 
180684 
££6£16 
155708 
361740 
000931 
41£5£3 
36£648 
l£69a9 
453489 
0141£8 
403£57 
434£61 



-0. 031 123 
-0. 369761 
-0. 000931 
-0. 13£300 

-0. 493835 
-0. 449965 
-0. 489346 
-0. 160684 
-0. ££6£16 
-0. 155708 
-0. 3£604£ 
-0. 000931 
-0. 345£44 
-0. 319956 
-0. 146510 

-0.41£105 
-0. 016507 
-0. 3579£6 
-0, 38574S 



ur 

1.000000 
0. 9a7£40 

1.000000 

0.961616 
0. 990989 
0. 983£91 
0. 964£7£ 

1.000000 
1 . 000000 
1.000000 
0. 988S91 

1.000000 

0. 976975 
0. 986514 
0. 991070 
0. 983£40 
0. 999903 
0. 98440£ 
0. 96£510 



P > .05 



0. 000000 
1.669537 

0. 000000 

0. 591 £90 
-0. 45649£ 
1.3£0095 
0. 844£57 
0. 000000 
0. 000000 

0. 000000 

-1.043553 
0. 000000 
-1.406690 
-I. 159334 
0. 6£5347 
-1.047251 
0. 7£4916 
-1. 160001 
-I. 184994 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 

S: Similarity Measure: II » Cosine. #2 « Dice 

TU: Term Weighting Schemes: See Result Page R-1 *' 

Correlation Coefficients: rjg Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (I • most relevant. 
4 • most non-relevant) a strong negative correlation shows agreenent 
between user's and system's relevance judgments. 

Significance Level: A positive Z 1i;d1cates that the second correlation Is higher 
than the first correlation (r4^> rju). If this Z Is statistically 
significant as Indicated by the asterisks, then resolving anaohors Improves 
tiii i^ttem's predications of relevance. 262 
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Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 

CENTRAL PRONOUNS 
Correlation Coefficients Significance L#v#l 

^ ^ '•ju '•jr ''ur 2 P > .05 
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-0. £87590 


0. 969677 
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NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: II « tosine. 12 « Dice 
TH: Term Weighting Schemes: See Result Page R-l 

Correlation Coefficients: rjy is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. r^r is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from lew to high (1 - most relevant, 
J ■ most non-relevant) a strono negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z indicates that the second correlation is higher 
than the first correlation (r4^> rj„). If this Z is statistically 
significant as indicated by the asterisks, then resolving anaphors improves 
tht system's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



CENTRAL PRONOUNS 



Correlation Coefficients 



Significance Levgl 
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-0. 000931 
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-0. 000931 
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0. £4499£ 
0. 000861 
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'0. 000931 
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0. 006594 
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-0. 000931 
0. 154452 
0. 233146 
0.025611 
0. 269456 
0. 000946 
0. 275464 
0. 309959 



ur 

1.000000 

0. 974837 

1.000000 

0.996858 
0. 972531 
0. 972810 
0. 975209 
1.000000 

1.000000 
1.000000 

0. 984463 
1.000000 
0. 983097 
0. 983441 
0. 999946 
0. 991456 

1.000000 

0. 988087 
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P > .05 



0. 000000 
0. 128531 
0. 000000 
-0. 765583 
-0.313510 
0.019795 
-0. 156356 

0. 000000 

0. 000000 
0. 000000 
-0. 750896 
0. 000000 
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-0. 876884 
-0. 645404 

-0.818913 
-0.415749 
-0. 776200 
-0. 581924 



NOTES: 

Q: Queries 100-199 were searched on INSPEC; 200-299 on PsychlNFO 
S: Similarity Measure: II » Cosine. #2 » Dice 
TM: Term Weighting Schemes: See Result Page R-1 



Correlation Coefficients: rju Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rir Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 • most relevant, 
4 • most non»relevant) a strona negative correlation shows agreeiient 
between user's and system's relevance Judgments. 

Significance Level: A positive Z Indicates that the second correlation 1$ higher 
than the first correlation (rjp> r^^,), if this Z Is statistically 

f2"lIIf!IIf " by tfie asterisks, then resolving anaphors Improves 

the fyttem's predications of relevance. 
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Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



NOMINAL DEMONSTRATIVES 



Correlation Coefficients Significance Level 



0 




TH 




r > 


r 

ur 


2 


203 


I 


m 


-0. 105343 


-0. 106148 


0. 999960 


0. 553664 


£03 




d 


-0. 141626 


-0. 160415 


0. 995277 


0. 851457 


203 




• 


-0.001022 


-0.001022 


1.000000 


0. 000000 


203 




h 


0. 117064 


0. 104833 


0. 999526 


1 . 737331 


203 




J 


0. 072857 


0. 094659 


0. 993378 


-0. 828564 


203 




HI 


-0. 097276 


-0. 112616 


0. 993226 


0. 577519 


203 




n 


-0.012827 


-0. 012665 


0. 992025 


-0. 005580 


203 


2 


a 


-0. 042S74 


-0.043710 


0. 999964 


0. 582901 


203 


2 


b 


0. 005796 


0. 005370 


0. 999995 


0. 574551 


203 


2 


c 


-0i. 037613 


-0. 036579 


0. 999974 


0.565132 


203 


2 


d 


-0. 070321 


-0. 081952 


0. 997292 


0. 690755 


203 


2 


• 


-0. 001022 


-0.001022 


1.000000 


0. 000000 


203 


2 


f 


-0. 036263 


-0. 050291 


0. 997751 


0.912521 


203 


2 


S 


-0. 070989 


-0.083169 


0. 997262 


0. 719449 


203 


2 


h 


0.069005 


0. 065814 


0. 999924 


1. 129174 


203 


2 


J 


0. 034705 


0. 026458 


0. 997600 


0.519096 


203 


2 


1 


0.002329 


0. 002250 


1.000000 


0. 422963 


203 


2 


m 


-0. 044468 


-0. 059333 


0. 996541 


0. 760132 


203 


2 


n 


0. 004985 


-0. 003015 


0. 996376 


0. 409634 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 « Cosine. 12 « Dice 
TH: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rjr is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 » most relevant, 
4 ■ most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation is higher 
than the first corrtlatlom (rir> rju). If this Z Is sUtistically 
significant as Indicated by tne asterisks, then resolving anaphors improves 
the system's prtdicatlons of relevance. 
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Page 

A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 



NOMIHAL DEMOHSTRATIVES 



Correlation Coefficients Significance level 



Q 


s 


TV 


ju 




r 

ur 


Z 




1 


a 


0. 089728 


0. 088733 


0. 999863 


0. 269908 


207 


1 


d 


-0. 006084 


0.016294 


0. 972406 


-0. 42S710 


207 


I 


* 


>0. 0009S8 


-0.000956 


i.ooooeo 


0. O0O000 


207 


1 


h 


0.213822 


0.214531 


0.999991 


-0. 758070 


207 


I 


J 


0. 168212 


0. 160671 


0.999335 


0.935811 


207 


i 


Nl 


0. 0991 18 


0. 099707 


0.993702 


-0. 023563 


207 


i 


n 


0. 06485S 


0.063666 


0.991169 


0. 040086 


207 


2 


m 


0.069602 


0.068528 


0.999717 


O. 202447 


207 


2 


b 


0. 054739 


0. 054262 


0. 999936 


0. 188977 


207 


2 


c 


0.070263 


0.069245 


0.999769 


0.212202 


207 


2 


d 


0. 021932 


0. 057765 


0.975297 


-0. 721826 


207 


2 


• 


-0.000956 


-0.000956 


1.0OOOO0 


0. 000000 


207 


2 


f 


-0. 040281 


0. 018031 


0.959633 


-0.916990 


207 


2 


8 


0. 018805 


0. 058379 


0. 974192 


-0. 779983 


207 


2 


h 


0. 202720 


0. 203888 


0.999983 


-0. 909808 


207 


2 


J 


0. 193024 


0. 193242 


0. 999661 


-0. 038158 


207 


2 


1 


0. 175987 


0. 179723 


0. 999665 


-0. 655097 


£07 


2 


IN 


0. 121846 


0. 137139 


0.994319 


-0. 646824 


207 


2 


• 

n 


0. 098820 


0. 117626 


0. 993677 


-0. 752024 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: II - Cosine. #2 « Dice 
TH: Teni Weighting ScheMs: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance Judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 • most relevant, 
4 • most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation lrw> rj^^), If this Z Is sUtlstlcally 
significant as Indicated fcy tne asterisks, then resolving anaphors Improves 
the system's predications of relevance. _ 
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Page 

A Statistical Coaparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 

NOMINAL DEMONSTRATIVES 



Correlation Coefficients 



Significance Level 



Q 


S 




r. 


r 

ur 


z 




1 


a 


>e. ?.32393 


-9. 534532 


9. 999965 


1.423618 




1 


d 


-e. 630979 


-9. 699182 


9.999699 


-9. 972057 


BIZ 


1 


• 


-0.281644 


-9.367326 


9. 976589 


2. 003152 


Ziz 


1 


h 


9. 089775 


9.992429 


9.999949 


-1.286973 




i 


J 


-9. 592539 


-9.564368 


9.994148 


-1.491171 


212 


1 


n 


-9. 663788 


-9.641139 


9.992193 


-1. 133026 


212 


1 


n 


-9. 791485 


-9.686994 


9.996452 


-1. 132692 


212 


2 


« 


'9. 553674 


-9.556193 


9.999956 


1.510422 


212 


2 


b 


-9.536118 


-9. 537247 


9.999999 


1.407011 


212 


2 


c 


-9. 569978 


-9. 562278 


9.999967 


1.525915 


212 


2 


d 


-9. 683127 


-9.699941 


9. 968614 


-1.737246 


212 


2 


• 


-9.28S411 


-9.498138 


9.969691 


2. 440264 


212 


£ 


f 


-9. 681729 


-9.696958 


9.969859 


-1.625723 


212 


2 


B 


-9. 692679 


-9. 616398 


9.966942 


-1.751584 


212 


2: 


h 


9.955268 


9.956689 


9.999967 


-0. 853978 


212 


£ 


J 


-9. 685346 


-9.649357 


9. 981874 


-1.459859 


212 


2 


1 


9. 994343 


9. 994597 


1.999909 


-0. 820542 


212 


2 


m 


-9. 719998 


-9. 652988 


9. 975446 


-1.623423 


212 


2 


V» 


-9. 731 159 


-9. 799484 


9. 999186 


-1.435622 



P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Slarflarlty Measure: #1 - Cosine. 12 - Dice 
TU: Tern Weighting Schencs: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance Judgnent and the 
systen's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgnent and the systen's predicted relevance based 
on resolved anaphors. 

Because the user's judgnents were scaled fron low to high (1 • nost relevant, 
4 • nost non-relevant) a strong negative correlation shows agrecnent 
between user's and systen's re1evan<:c judgnents. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (r4f.> rju). If this Z Is sUtlstlcally 
significant as Indicated by tne asterisks, then resolving anaphors Inproves 
the systen's predications of relevance. 
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Page 

A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgiients with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 



NOMINAL DEMONSTRATIVES 



Correlation Coefficients Significance Level 



Q 


S 


TM r. 






z 








JU 




ur 




213 


1 




-0. 331415 


-0. 331415 


1.000000 


0. 000000 


£19 




d 


-0. 431 196 


-0. 4£8167 


0. 96£5££ 


-0. 049107 


ai9 




• 


-0.001101 


-0.001101 


1.000000 


0. 000000 


£19 




h 


-0. 738016 


-0. 7£41S7 


0. 959347 


-0.£8657£ 


£19 




J 


-0.671656 


-0. 654470 


0. 960£16 


-0. 459517 


£19 




M 


-0,541866 


-0.537656 


0.971903 


-0. 084605 


£19 




n 


-0.56£968 


-0. 551813 


0.974578 


-0. £3864£ 


£19 


£ 




-0. £531£3 


-0. £S31£3 


1.000000 


0. 000000 


£19 


£ 


b 


-0. ££33££ 


-0.££33££ 


1.000000 


0. 000000 


£19 


£ 


c 


-0. £64359 


-0. £64359 


1.000000 


0. 000000 


£19 


£ 


d 


-0. 395491 


-0.379081 


0. 959384 


-0. £50061 


£19 


£ 


• 


-0.001101 


-0.001101 


1 • 009000 


0. 000000 


£19 


£ 


f 


-0. 380616 


-0. 357337 


0. 956167 


-0. 338536 


£19 


£ 


9 


-0. 404803 


-0. 387454 


0. 959993 


-0. 267448 


£19 


£ 


h 


-0. 769917 


-0.74£lie 


0.958057 


-0. S84SS1 


£19 


£ 


J 


-0. 594641 


-0. 6047£5 


0. 974991 


0. ££5788 


£19 


£ 


1 


-0.616435 


-0. 637544 


0. 973718 


0. 47101£ 


£19 


£ 


HI 


-0. 504693 


-0. 499509 


0. 9706£6 


-0. 099139 


£19 


£ 


n 


-0. 5£04ie 


-0. 514£££ 


0. 97£6S8 


-0. 1£4054 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Sliillarlty Measure: #1 • Cosine. #2 * Dice 
TU: Tera Weighting Scheics: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rw Is between 
the user's relevance judgtnt and the system's predicted relevance based 
on resolved amphors. 

Because the user's judgMnts were scaled from low to high (1 « most relevant, 
4 • aost non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance Judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than tJie first correlatloii {rwy rjy). If this Z Is sutlstlcally 
signff leant as Indicated by tne asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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Pige 

A Statistical Coaparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 



NOMINAL DEMONSTRATIVES 



Correlation Coefficients 



Significance Level 



Q 


s 


TW 




r . 


ur 


Z 






m 




O 04A756 




0. 071 199 


221 


1 


d 


a. a754e9 


-0.001118 


0. 976299 


1.318525 


221 


1 


• 


-e. eeaass 


-0.000685 


1.OOOO0O 


0. 000000 


221 


1 


h 


-0. 034912 


-0. 042102 


0. 999526 


0. 874369 


221 


1 


J 


-0. 072548 


-0. 111765 


0. 965440 


0.560721 


221 


1 


m 


0. 012463 


-O. 050746 


0.972265 


1.006163 


221 


1 


n 


-0. 082838 


-0. 124017 


0.973413 


0. 671893 


221 


2 


a 


0. 035589 


0. 035454 


0. 999933 


0. 037527 


221 


2 




0. 024068 


0. 023727 


0. 999973 


0. 173684 


221 


2 


c 


0. 035314 


0. 035149 


0. 999942 


0. 057249 


221 


2 


d 


0.040988 


-0.042582 


0. 966224 


1.206113 


221 


2 


• 


-0.000885 


-0.000885 


1.000000 


0. 000000 


221 


2 


f 


-0. 003799 


-0.069580 


0. 970821 


1.021014 


221 


2 


g 


0. 035177 


-0. 048160 


0. 964862 


1. 179566 


221 


2 


h 


-0. 034210 


-0. 038443 


0. 999778 


0.751501 


221 


2 


J 


-0. 086140 


-0. 115411 


0. 969342 


0. 747931 


221 


2 


1 


-0. 001531 


-0. 001531 


1.000000 


0. 000000 


221 


2 


M 


-0. 013498 


-0. 079919 


0. 967568 


0. 978256 


221 


2 


• n 


-0. 092854 


-0. 135324 


0. 973427 


0. 69391 1 



p > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 

S: Sliillarlty Measure: II * Cosine. #2 > Dice 

TU: Teni Weighting Schemes: See Result Page R-1 ** 

Correlation Coefficients: rjy Is between the user's relevance judgment and the 
systM's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgnent and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 • most relevant, 
4 ■ most nonrrelevant) a strong negative correlation shows agreement 
between user's and systaa's relevance judgments. 

Significance Level: A positive I Indicates that the second correlation Is higher 
than the first correlation (r4y.> ^ju)* this Z Is sutlstlcally 
significant as Indicated bf tne asterisks, then resolving anaphors Improves 
tke lyrstio's predications of relevance. 
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Appendix £-126 
Page 

A Statistical Coaparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgiients with Resolved 
Anaphors and User*s Relevance Judgnents: for Anaphoric Class 

NOMINAL DEMOKSTRATIVES 



Correlation Coefficients 



Significance Level 



0 


< 




jr 


222 


1 




065303 


-0. 065303 


£22 


1 


d 


-0. 407137 


-0. 362235 


222 


1 


• 


-0.000864 


-0.000684 


222 


1 


h 


-0. 162421 


-0. 180458 


222 


1 


J 


-0. 403934 


-0. 384673 


222 


1 


M 


-0. 430564 


-0. 392961 


222 


1 


r» 


-0. 194060 


-0. 191052 


222 


2 


m 


"9. 076259 


-0. 076259 


222 


2 


b 


-0. 063133 


-0. 063133 


222 


2 


c 


-0. 073564 


-0. 073564 


222 


2 


d 


-0.410646 


-0.366092 


222 


2 


• 


-0.000664 


-0. 000684 


222 


2 


f 


-0.338840 


-0.291939 


222 


2 


u 


-0.410561 


-0. 363829 


222 


2 


h 


-0. 022308 


-0. 022041 


222 


2 


J 


-0. 430670 


-0. 395590 


222 


2 


1 


-0. 002454 


002434 


222 


2 


m 


-0. 442443 


-0. 399871 


222 


2 


* ri 


-0. 256665 


-0.242209 



ur 

1.000000 
0.966419 
1.000000 
0. 999724 
0.986290 
0. 971649 
0. 990843 
1.000000 
1.000000 

1.000000 

0.971400 
1.000000 
0. 956741 
0. 966927 
0. 999997 
0. 976939 
1.000000 
0. 971535 
0.969115 



P > .OS 



0. 000000 
-0. 834665 

0. 000000 

-0. 379519 
-0. 564383 
-0. 770674 
-0. 101972 
0. 000000 
0. 000000 
0. 000000 
-0.901713 
0. 000000 
-0. 749653 
-0. 676164 
-0. 529832 
-0. 8332S1 
-0. 223601 
-0. 672841 
-0. 458413 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Slirilarlty Measure: II « Cosine. 12 > Dice 
TU: Tern Weighting Schenes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance judgment and the 
systen's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance Judgwnt and the systcn's predicted relevance based 
on resolved aniphprs. 

Because the user's judgments were scaled from low to high (1 « most relevant, 
4 » most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation {r^y rju). If this Z Is sUtlstlcally 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
tilt mtai's predications of relevance. 
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Page 

A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgaents with Resolved 
Anaphors and User's Relevance Judgaents: for Anaphoric Class 

NOMINAL DEMONSTRATIVES 



Correlation Coefficients 



Significance Level 



TU 



ur 



P > .05 



SS3 
223 
223 
223 
223 
223 
223 
223 
223 
223 
223 
223 
223 
223 
223 
223 
223 
223 
223 



2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 



• -©.732594 
d -0.694873 

• ""^4^ 0006^0 
h -0.273036 
J -0.422509 
M -0.695676 
n -0.639629 

• -0.754136 
b -0.718551 
c -0.752835 
d -0.732325 



f -0.709220 
g -0.73184S 
h -0.361158 
J -0.389875 
1 -0.315803 
M -0.726692 
n -0.670259 



-0. 732594 
-0. 673473 
-0. 000648 
-0. 373048 
-0.421755 
-0. 674331 
-0. 612636 
-0.754136 
-0. 718551 
-0.752835 
-0.670997 
-0.000648 
-0. 631819 
-0.667450 
-0.361167 
-0. 391874 
-0. 315787 
69&03& 
-0.634363 



1. 

0. 991966 
1.000000 

1.000000 

0.999917 
0. 994234 
0. 993787 
1.000000 
1.000000 
1.000000 
0. 978573 

O. 972828 
0. 977200 
1.000000 
O. 999818 
1.000000 
O. 982395 
0. 960962 



O. 000000 
-0. 836911 

o. oooooo 

O. 364864 
-0.241072 
-O. 972685 
-1.110079 

O. OOOOOO 

0. oooooo 
o. oooooo 

-1. 397333 
O. OOOOOO 
-1.497024 
-1.415382 
O. 326963 
0. 424475 
-O. 442164 
-0. 840189 
-O, 879345 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 

S.: Slallarlty Measure: II « Cosine. , 12 « Dice 

TU: Tern Uelghting Sche«es: See Result Page R-1 ' 

Correlation Coefficients: rjy Is between the user's relevance judgment and the 
systca's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgaent and the system's predicted relevance based 
on resolved anaphors. 

lecause the user's judgments were scaled from low to high (1 » most relevant, 
4 « aost non-relevant) • strong negative correlation shows agrecMont 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
tlian the first corrtlatlon (rir> ^ju)* this Z Is sUtlstlcally 
tffnff leant as Indicated ky tne asterisks, then resolving anaphors laproves 
the lystai's predications of relevance. 2*^^ 
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Page 

A Statistical CoMparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents wfth Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 



NOMINAL DEMONSTRATIVES 



Correlation Coefficients 



Significance Level 



Q 


S 


T« 




"jr 


''ur 


z 


PPT 


A 






1S6573 


1.000000 


0. 000000 


2:27 


i 


d 


-0. 321 133 


-0. 282110 


0.914091 


-0. 442966 


227 


I 


m 




IP» WVC90 


1.000000 


0. 000000 


227 


1 


h 


0. 333390 


0. 339230 


0. 990074 


~0. 196656 


227 


1 


J 


0. 059223 


0.091133 


0. 963962 


-0. 533222 


227 


1 


M 


-0. 227812 


-0. 182803 


0. 923801 


-0. 527222 


227 


1 


n 


*0. 061523 


-0. 0S8S20 


0. 942956 


-0. 039835 
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p > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200>299 on PsychlNFO 
S: Sinllarlty Measure: #1 * Cosine. 12 « Dice 
TV: Teni Weighting Schcaes: See Result Page R-1 

Correlation Coefficients: rju between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. r]r between 
the user's relevance judgaent and the system's predicted relevance based 
on resolved anaphors. 

Because the user's JudgMtnts were scaled fron low to high (1 « nost relevant, 
4 * MSt non-relevant) a strong negative correlation shows agreenent 
between user's and syste«*s relevance judgnents. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
ABU the first correlation (rir> rj^). If this Z Is sutlstlcally 
significant as Indicated by tne asterisks, then resolving anaphors Inproves 
tlw lyitMi'i predlcatlow of relevance. 
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Appendix E-129 
Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 

NOMINAL DEMONSTRATIVES 



Correlation Coefficients 



Significance Level 
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.05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: II - Cosine. #2 - Dice 
TW: Term Weighting Schemes: See Result Page R-1 ' 

Is between the user's relevance judgment and the 



Correlation Coefficients: 
system's predicted re1e< 
the user's relevance 
on resolved anaphors. 



rju Is between the user's relevance judgment and the 
levance based on unresolved anaphors. rjr Is between 
judgment and the system's predicted relevance based 



Because the user's judgments were scaled from low to high (1 • most relevant, 
4 • iaost non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation lrw> rju). If this Z Is sUtlstlcally 
significant as Indicated fay the asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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Page 

A Statistical CoMparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 



NOMINAL DEMONSTRATIVES 



Correlation Coefficients 



Significance Level 
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p > .05 



<»*•» 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: SiMllarlty Measure: #1 - Cosine. #2 « Dice 
TH: Tern Weighting ScheMs: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. 



the user's relevance judgment and the systmi's predicted rel 
on resolved anaphors. 



rir Is between 
evence based 



Because the user's judgments were scaled from low to high (1 • most relevant, 
4 • most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (r4r> Vju). If this Z Is statistically 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 



NOMINAL DEMONSTRATIVES 



Correlation Coefficients 



Significance Level 
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-0. 399141 
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p > .05 



<»«»« 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Slallarlty Measure: #1 « Cosine. 12 • pice 
TV: Tsra Weighting Scheaes: See Result Page R-1 

Correlation Coefficients: rj^. Is betweeii the user's relevance judgment and the 
systcn's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgwnt and the systcn's predicted relevance based 
on resolved anaphors. 

Because the user's judgaents were scaled fron low to high (1 • nost relevant. 
4 « Host non-relevant) a strong negative correlation shows agrecnent 
between user's and system's relevance Judgnents. 

Slgniflcanct Level: A positive Z Indicates that the second correlation Is higher 
than the first correUtloii {nr^ rju), If this Z Is statistically 
significant as Indicated tne asterisks, then resolving anaphors laproves 
tilt «yst«M*s predications of relevance. 
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Page 



Unreso 
Anapho 



A Statistical Comparison of the Relationship Between 
Ived Anaphors and User's Relevance Judgments with Res 
rs and User's Relevance Judgments: for Anaphoric Cla 



NOMINAL DEMONSTRATIVES 
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Correlation Coefficients 



Significance Level 
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0. 995845 


-0. 878531 
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P > f05 



NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 • Cosine. #2 « Dice 
TV: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rj^ Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rw Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved aiMphors. 

Because the user's judgmoHts were scaled from low to high (2 « most relevant, 
4 ' most noiMTf levant) a strong negative correlation shows agreement 
between user's and system's relevance Judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rj,.> rjy). If this Z Is stetlstlcally 
ttanlf leant as Indicated by the asterisks, then resolving anaphors Improves 
tht system's predlcatlow of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



RELATIVE PRONOUNS 



Correlation Coefficients 
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NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: II * Cosine. 12 « Dice 
TW: Term Weighting Schemest See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's rt1evanc« Judgment and tht system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 « most relevant, 
4 • most non-relevant) a strong negative correlation shows agreement 
between user's and system's r«1evance Judgments. 

Significance Level: A posHfve Z IndlcaUs that the second correlation Is higher 
than the first correlation {rw> rju). If this Z Is statistically 
sfgnff leant as Indicated by tne asterisks, then resolving anaphors Improves 
the i^tom's prtdlcatlons of relevance. ^^.^ 
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Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



RELATIVE PRONOUNS 



Correlation Coefficients Significance Level 
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NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: 11 • Cosine. #2 » Dice 
TU: Term Nelghtlng Schemes: See Result Page R-1 

Correlation Coefficients: rju ^* between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's Judgments were scaled from low to high (1 « most relevant, 
4 • most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rjy.> rju)* I Is sutlstlcally 

significant as Indicated by the asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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A Statistical CoMparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 

RELATIVE PRONOUNS 



Correlation Coefficients 
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p > .05 
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NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 « Cosine. 12 « Dice 

TV: Term Uclghtfng Schemes: See Result Page R-1 

Correlation Coefficients: 

I 

judgment and the system's predicted 



elation Coefficients: rju Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance base4 



on resolved anaphors. 



Because tlie user's judgments were scaled from low to high (1 « most relevant, 
4 • most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance level: A positive Z indicates that the second correlation Is higher 
than Urn first correletion {rw> rju)* this Z is statistically 
significant as indicated Ijf the asterisks, then resolving anaphors Improves 
the lixstea's predic«ti9ns of relevance. 
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A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



RELATIVE PRONOUNS 



Correlation Coefficients S1<in1f Icance level 
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NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: SlalUrlty Measure: II « Cosine. #2 « Dice 
TU: Teni Weighting Schcaes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance judgtient and the 
syst«*$ predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the systcn's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 « most relevant, 
4 • most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level; A positive Z Indicates that the second correlation Is higher 
than the first correlation ir<r> rju). If this 7. Is ^utlstlcally 
significant as Indicated by tfi^ asterisks, then resolving anaphors Improves 
tilt system's prtdlcetlons of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



RELATIVE PRONOUNS 



Correlation Coefficients Significance Level 



A 
V 


5 


TV 








7 










"^ur 




asi 


1 


a 


0. 044923 


0. 044756 


0. 999961 


0.071199 






d 


0. 075409 


0. 0S34S1 


0. 986470 


1. 184675 


SSI 




• 


-0. 000885 


•-0. 000885 


1.000000 


0. 000000 


SSI 




h 


-.0. 03491S 


-0. 04S933 


0.999682 


1. 190980 


SSI 




J 


-0. 07S548 


-0. 111107 


0. 987392 


0.91SS37 


SSI 




Nl 


0. 01S463 


-0. 033502 


0. 986504 


1.047693 


SSI 




n 


-0. 08S83a 


-0. 117352 


0. 988757 


0. 865S58 


SSI 


s 


« 


0. 03SS69 


0. 035454 


0.999933 


0. 0375S7 


SSI 


s 


b 


0. 024088 


0. 023727 


0. 999973 


0. 173664 


SSI 


s 


c 


0. 035314 


0^035149 


0. 999S -2 


0. 0S7S49 


SSI 


s 


d 


0.040988 


-0. 023433 


0. 981S67 


1.S47183 


SSI 


s 


• 


-0. 000885 


-0. 000885 


1.000000 


0. 000000 


SSI 


s 


f 


-0. 003799 


-0. 059320 


0. 985956 


1.S413S3 


SSI 


s 


g 


0. 035177 


-0. 030448 


0. 980937 


1.S59488 


SSI 


s 


h 


-0. 034S10 


-0. 039901 


0.999854 


1.S44787 


SSI 


s 


J 


-0. 068140 


~0. 107814 


0. 987380 


0. 985108 


SSI 


s 


1 


-0. 001531 


-0.001531 


1.000000 


0. 000000 


SSI 


s 


m 


-0.013498 


-0. 065346 


0. 985043 


1. 1S3404 


SSI 


2 


* n 


-0. 09S854 


-0. 128436 


0.989195 


0.910719 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 * Cosine. #2 « Dice 
TV: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 « most relevant, 
4 ■ most nofi-rtlevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rj|.> rjy). If this Z Is statistically 
significant as indicated by tne asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 



RELATIVE PRONOUNS 



Correlation Coefficients Significance Level 
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NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Sliillarlty Measure: #1 « Cosine. #2 - Dice 
TU: Tern Uelghting SdicMS: See Result Page R-1 

Correlation Coefficients: rjy 1$ between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 « most relevant, 
4 • most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A voosltfve Z Indicates that tte second correlation Is higher 
thaii the first correlation Ktwy rju). If this Z Is sutlstlcally 
sf gulf leant as Indicated by tne asterisks, then resolving anaphors Improves 
tht system's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



R ELATIVE PRONOUNS 



Correlation Coefficients Significance level 
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NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-259 on PsychlNFO 
S: Similarity Measure: #1 « Cosine. «2 « Dice 
TU: Term Ueighting Schemes: See Result Page R-1 

Correlation Coefficients: rju is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 « most relevant, 
4 • most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation is higher 
than tlie first correlation (rjr> i^ju)- ^^is Z is sutistically 
significant as Indicated fey tne asterisks, then resolving anaphors improves 
the liysteB's predications of relevance. 
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A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 

RELATIVE PRONOUNS 

Correlation Coefficients Significance Level 
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NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 « Cosine. «2 « Dice 
TW: Te. . Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance Judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's Judgments were scaled from low to high (1 * most relevant. 
4 * most non-relevant) a strong negative correlation shows agreement 
between user's and sj^te«*s relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correletlor. (rj(.> rjy). If this Z Is statistically 
significant 4$ indicated by the asterisks, then resolving anaphors Improves 
the ^stem's predications of relevance. 
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A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgaents: for Anaphoric Class 



RELATIVE PRONOUNS 



Correlation Coefficients 
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NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Slall an ty Measure: #1 « Cosine. #2 « Dice 
TU: Teni Weighting Scheaes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance judgment and the 
systea's predicted relevance based on unresolved anaphors. r^r between 
the user's relevance judgaent and the systea's predicted releVance based 
on resolved anaphors. 

Because the user's Judgaents were scaled froa low to high (1 « aost relevant. 
4 « aost non-relevant) a strong negative correlation shows agree^t 
between user's and systea's relevance Judgaents. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rjr> rjy). If this Z Is sUtlstlcally 
significant as Indicated by tne asterisks, then resolving anaphors laproves 
the nysiM's predlcetlons of relevance. 
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A Statistical Coiiparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



RELATIVE PRONOUNS 



Correlation Coefficients 
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NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Sinllarlty Measure: #1 « Cosine. #2 « Dice 
TU: Teni Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rjy is between the user's relevance judgnent and the 
system's predicted relevance based on unresolved anaphors. rj^ Is between 
the user's relevance Judgment and the system's predicted relevance ba^«d 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 « most relevant, 
4 « most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance Judgments. / 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rjr> rj^)* If this Z Is statistically 
significant at Indicated bf tne asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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Appendix E-143 
Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgiients with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 

RELATIVE PRONOUNS 



Correlation Coefficients 



ur 

e. 933898 
0. 940601 
1.000000 
1.000000 

0. 9983&S 
0. 978 15S 
0. 987527 
0. 993855 
0. 333989 
0. 993882 
0. 944130 

1.000000 

0. 934141 
0. 940619 
1.000000 

0. 998910 

1.000000 

0. 373895 
0. 986337 



Q 


s 


TW r. 










JU 




c^hQ 


1 






-0. 171062 




1 


c 


-0. 376824 


-0. 337026 




1 




-0. 001000 


-0.C B1000 




1 


h 


-0.209710 


^0. 209649 


248 




J 


-0. 240205 


-0. 233500 


248 


1 


m 


-0. 384995 


-0. 360608 


243 


1 


ri 


-0.315971 


-0. 236292 


248 


2 


a 


-0. 162555 


-0. 158754 


248 


2 


b 


-0. 162328 


-0. 160633 


246 


2 


c 


-0. 165652 


^0. 162231 


248 


2 


d 


-0. 368644 


-0. 366638 


248 


2 


• 


-0.001000 


-^0. 001000 


248 


2 


f 


-0.387811 


-0. 343652 


248 


2 


S 


-0. 365112 


-0. 362132 


246 


2 




-0. 209574 


-0. 209543 


248 


2 


J 


-0. 253679 


-0. 247233 


248 


2 


1 


-0. 206419 


-0. 206393 


248 


2 


m 


-0. 426066 


-0. 414015 


240 


2 


• 

n 


-0. 346954 


-0. 337393 



Significance Level 



-0. 990842 
-0. 553664 

0. 000000 
-1-213183 
-0. 539117 
-0. 561105 
-0. 584208 
-1.010335 
-0. 978255 
-1.006056 
-0. 318389 

0. 000000 
-0. 503549 
-0. 321994 
-0. 862734 

-0. 630726 
-0. 627168 
-0. 260055 
-0. 332816 



p > .05 



NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 » Cosine. f^2 > Dice 
TW: Term Weighting Schemes: See Result Page R-l 

Correlation Coefficients: rjy Is between the user's relevance judgii»nt and the 
system's predicted relevance based on unresolved anaphors. Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 « most relevant, 
4 » most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than tl)e first correlation (rjr> rjy). If this Z 1$ statistically 
significant as Indicated by the asterisks, then resolving anaphors Improves 
the <yft«i*s predications of relevance. 
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Appendix E-144 



Page 

A SUtlstlcal Conparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgnentse for Anaphoric Class 



RELATIVE PROWOUWS 



Correlation Coefficients 



Significance Level 



TM 



S5£ 



£52 
£S2 
£52 
£52 
252 
252 
252 
252 
252 
252 
252 
252 
252 
£52 
252 



1 
1 

4 

2 
2 
2 



2 
2 
2 
2 
2 
2 
2 



a 0 

h 

J 
rn> 
ri 
a 



0. 

e. 
e. 

0. 
0. 
b -0. 
c 0. 
d 0. 
• -0. 

f 0. 

g 0. 
h 0. 

J 0. 

I 0. 

m 0. 
ri 0. 



. 087821 
. 146321 
000931 
167661 
203856 
200105 
253456 
006594 
029094 
009133 
188045 
000931 
035067 
196326 
024025 
244992 
000661 
248129 
295229 



0. 087621 
0.261C77 

-0. 000931 
0.214784 
0. 322690 
0. 294093 
0. 30301 1 
0. 006594 

-0. 029094 
0. 009133 
0. 233418 

-0. 000931 
0. 093920 
0. 231460 
0. 028344 
0. 297828 
0.001161 
0. 288191 
0.316754 



ur 

1.000000 

0. 968427 
1.000000 
0. 997835 
0. 974105 
0. 979743 
0. 991606 
1.000000 
1.00000e 

1.000000 

0. 989041 
1.000000 
0.993518 
0. 990525 
0. 999957 
0. 991662 
1.000000 
0. 992227 
0. 995733 



0. 000000 

-1.962495 
0. 000000 
-1.766706 
-2. 137546 
-2. 003274 
-1.660929 
0. 000000 
0. 000000 
0. 000000 
"1.322697 
0. 000000 
0. 042944 
-1. 104512 
-1.978265 
-1.770730 
-1.939500 
-i. 399396 
-1.^930039 



P > .05 



<«**« 



<*»*« 
<•«*» 



NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNR) 
S: Slnliarlty Measure: II - Cosine. #2 « Dice 
TW: Tern Weighting Scheaws: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance judgment and the 
systtn's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 * most relevant, 
4 • nost non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rjr> rju). If this Z Is statistically 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the «ytteti^s predications of relevance. 
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Appendix E-145 
Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



NOMINAL SUBSTITUTES 



Correlation Coefficients Significance Level 



Q 


S 






''ur 


z 


£213 


1 


a 


-0. 105343 


-0. 106148 


0. 999980 


0. 553864 


£03 


1 


d 


-0. 141&£6 


-0. 1££017 


0. 997337 


-1. 179376 


£03 


1 


• 


-0.0010££ 


-0.e0i0££ 


1.000000 


0. 000000 


£03 


i 


h 


0. 1170£4 


0. 1£1139 


0. 999748 


-0. 797060 


£03 


1 


J 


0. 07£a57 


0. 097196 


0. 996786 


327002 


£03 


« 


m 


-0. 097£7S 


-0. 075291 


0. 996669 


-1. 177638 


£03 


1 


n 


-0. 01£S£7 


0. 007 19£ 


0. 997692 


-1.284602 


£03 


£ 


a 


-0. 04£574 


-0. 043710 


0. 999964 


0.582901 


£03 


£ 


b 


0. 005796 


0. 005370 


0. 999995 


0.574551 


£03 


£ 


c 


-0. 037&13 


-0. 038579 


0. 999974 


0. 585132 


£03 


£ 


d 


-0. 0703£1 


-0. 047237 


0. 996690 


-1.238604 


£03 


£ 


• 


-0. 001 0££ 


-0. 001022 


1.000000 


0. 000000 


£03 


£ 


f 


-0. 03&£&3 


-0.019334 


0. 997846 


-1. 124668 


£03 


£ 


o 


-0. 070989 


-0. 048683 


0. 99682S 


-1.222133 


£03 


£ 


h 


0. 069005 


0. 071266 


0. 999944 


-0.931962 


£03 


£ 


J 


0. 034705 


0. 05686£ 


0. 996951 


-1.237896 


£03 


£ 




0. 00£3£9 


0. 00£487 


1.000000 


-1. 196375 


£03 


£ 


Iff 


-0. 044468 


-0. 0£159& 


0. 996640 


-1. 21692£ 


£03 


£ 


. r, 


0. 004985 


0. 0£3496 


0. 998056 


-1. £94277 



P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: II - Cosine. #2 « Dice 
TW: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rj^ 1$ between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance Judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's Judgments were scaled from low to high (1 « most relevant, 
4 • most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance Judgments. 

Significance Level: A positive Z Indicates that the second correlation 1$ higher 
than the first correlation (r4r> rjy). If this Z Is statistically 
significant as Indicated bjf the asterisks, then resolving anapitors Improves 
the system's predications of relevance. » k r « 
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Appendix E-I46 
Page 

A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgaents: for Anaphoric Class 



NOHIhAL SUBSTITUTES 



Correlation Coefficients 



Significance Level 



£®7 
£07 
287 
£07 
£07 
£07 
£07 
£07 
207 
£07 
£07 
£07 
£07 
£07 
£07 
£07 
£07 
£07 
£07 



£ 
£ 
£ 
£ 
£ 
£ 
£ 
£ 
c 



TW 

A 0. 

d -0. 

• -0. 

h 0. 

J 0. 

m 0. 

n 0. 

a 0. 

b 0. 

c 0. 

a 0. 

• — 0. 
f -0. 
g 0. 
n 0. 

J 0. 

1 0. 

m 0. 

r, 0. 



0897£8 

006064 

000956 

£138££ 

16d£l£ 

099118 

06485S 

06960£ 

054739 

070£63 

0£193£ 

000956 

040£81 

016805 

£0£7£0 

1930£4 

175987 

1S1846 

0988£0 



0. 088733 
-0. 0150£1 
-0. 000956 
0. £13813 
0. 1663£4 
0. 094709 
0. 063781 
0. 0685£8 
0.054£6£ 
0. 069£45 
0. 013465 
-0. 000956 
-0. 045760 
0. 010348 
0. £0£736 
0. 19£399 
0. 1 76003 
0. 118406 
0. 098040 



ur 

0. 993863 
0. 99770£ 

1.000000 

1.000000 

0.999935 
0. 999233 
0. 999£68 
0. 999717 
0. 999936 
0. 999769 
0. 998701 

1.000000 

0. 999586 
0. 9988£7 

1.000000 

0. 999987 

1.000000 

0- 999763 
0. 999844 



P > .05 



0. £69908 
0. 59086£ 
0. 000000 
0. 95997£ 
0. 75£71£ 
0. 505890 
0. 1£573£ 
0. £0£447 
0. 188977 
0. £1££0£ 
0.743144 
0. 000000 
0. 6;:i£67£ 
0. 780997 

-0. 935803 
0. 565£0S 

-0. 6950£3 
0.711351 
0. 198758 



NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Slirflarlty Heasure: #1 * Cosine. #2 > Dice 
TW: Tern Weighting SchcMs: See Result Page R-l 

Correlation Coefficients: rjy Is between the user's relevance judgment and the 
systen's predicted relevance based on unresolved anaphors. rw Is between 
the user's relev&^ice JudgMnt and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 > most relevant, 
4 » most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level; A positive Z Indicates that the second correlation Is higher 
than the first correlation (ri|.> rjy). If this Z Is sUtlstlcally 
significant as Indicated by tne asterisks, then resolving anaphors li^roves 
the lystam's predications of relevance. 
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Appendix 



Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgaents: for Anaphoric Class 



RON IMAL SUBSTITUTES 



Correlation Coefficients 



Significance Level 



Q 


s 






£l2- 






-e. 532393 


-0. 534533 


£■!£ 


1 


d 


-0. 639979 


-0. 646427 


£1£ 




• 


-0. 281644 


-0. 326587 


£1£ 


1 


h 


0. #89775 


0. 049367 


£1£ 




J 


-0. S92S39 


-0. 616077 


£l£ 




m 


-0. 663788 


-0.677796 


£1£ 


1 


ri 


-0. 701483 


-0. 708855 


£12 


2 


a 


-0. 553674 


-0. 556193 




c 


b 


-0. 536118 


-0. 537247 


21£ 


2 


c 


-0.560078 


-0. 562278 


£1£ 


£ 


d 


-0. 683127 


-0. 683279 


212 


2 


• 


-0.288411 


-0. 320491 


212 


2 




-0. 681729 


-0. 682627 


212 


2 


5 


-0.692679 -0.692217 


£12 


2 




0. 055268 


0.031040 


£12 




w 


-0. £85346 


-0. 689855 


212 


2 


1 


0. 004343 


0. 002371 


212 


2 


m 


-0. 710908 


-0. 710451 


212 


2 


V> 


-0.731150 


-0. 728244 



IT 


2 


r ^ 


0.999965 


1.423618 




0. 998339 


1.592860 




0.984881 


1.317287 




0. 978870 


0. 965636 




0. 997809 


1.982061 


<**«* 


0. 998814 


1.732608 




0.999517 


1.517207 




0.999956 


1.510422 




0.999990 


1.407011 




0.999967 


1.525915 




0.997142 


0. 013529 




8.992082 


1.299060 




0.996382 


0. 070769 




O. 996861 


-0. 039660 




0.992915 


0. 998164 




0.999186 


0.741864 




0. 999954 


1.011592 




0.998167 


-0. 052548 




0.999230 


-0. 525755 





NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Slirilarlty Measure: II » Cosine. #2 « Dice 
TH: Teni Weighting SchcMs: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance Judgnent and the 
systcn's predicted relmnce based on unresolved anaphors. rjr Is between 
the user's relevance judgMnt and the systcn's predicted relevance based 
on resolved .anaphors. 

Because the user's judgMonts were scaled fro« low to high (1 * Mst relevant, 
4 « lost non-relevant) a strong negative correlation shows agreenent 
between user's and systen's relevance JudgMnts. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation {tw> rju). If this Z Is sUtlstlcally 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the «yst«a's predications of relevance. 
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Appendix E-148 
Page 

A Statistical CoMparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments if 1th Resolved 
Anaphors and User's Relevance Judgaents: for Anaphoric Class 

NOHIilAL SUBSTITUTES 



Correlation Coefficients 



Significance Level 



SI'S 
£1S 
219 
219 
£19 
219 
219 
219 
219 
219 
£19 
219 
£19 
219 
219 
£19 
219 
£19 
£19 



1 
2 



2 
2 



TH 



d 
• 

*> -9. 

3 

M — 

« -8. 
a 

tj -9. 
c -9. 
d -9. 
• -9. 
f -9. 
c —9. 

h -a. 

J 

1 -e. 

m -9. 
'r, —9. 



331415 
431196 
Miiiai 
73891£ 
£71^6 
S4186S 

se£9&a 

253123 
223322 
264359 
395491 

e(?ii0i 

380616 
4«4Bd3 
769917 
594641 
616435 
504693 
520418 



-0. 331415 
-0.475251 

-0. 001101 

-0. 753976 
-0.711427 
-0.581991 
-0.601222 
-0.253123 
-0.223322 
-0.264359 
-0.448579 
-0. 001101 
-0. 435879 
-0.458173 
-0.781529 
-0. 640988 
-0. 661934 
-0. 553519 
-0. 565979 



ur 

1^000000 

0.995865 

1.000000 

0.977907 
0.994462 
0.995766 
0.996420 

■A • WW^^W 
1.000000 
1.000000 

0. 993423 

0.992508 
0.993070 
0.978468 
0.993603 
0.985596 
0.993948 
0.995212 



0. 000000 

2. 001406 

0.454360 
1.776276 
1.877988 
1.941727 
0.000 0 00 
0.000000 
0.000000 

1.915945 

0.000000 

1.870053 
1.884412 
0. 355104 
1.823538 
1.305042 
1.889240 
1.966895 



P > .05 



<««•« 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: SiBllarUy Heasure: #1 » Cosine. 12 » Dice 
TH: Tera Weighting SchcMs: See Result Page R-1 

Correlation Coefficients: r*„ Is between the user's relevance judgnent and the 
systcn's predicted relevance based on unresolved anaphors. rjr Is between 
the user*s relevance judgMnt and the system's predicted relevance based 
on resolved amphors. 

Because the user's judgments were scaled fro« low to high (1 » Most relevant, 
4 » Nost non-relevant) a strong negative correlation shows agrecnent 
between user's and systen's relevance JudgMnts. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first corrtlatlon (r4|.> rju). If this Z Is sutlstlcally 
significant as Indicated the asterisks, then resolving anaphors iMproves 
the lystM's predications of relevance. 
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A Stetlstlcal Co«pir1soA of the Keletlonshlp Between 
Unresolved Anaphors and User's Relevance Judgaents with Resolved 
Anaphors and User's Relevance Judgaents: for Anaphoric Class 



NONINAL SUBSTITUTES 



Correlation Coefficients 



Q 


S 


TH r. 












Jtl 




ur 


2£1 


1 






V^^ZDo 


Wm 999961 


221 


,1 


d 


e. «7S4e>9 




99^47 


222 






-9.000885 




1«80MKM 


221 


1 


h 


-0. 034912 




V. 9986^6 


221 


I 


J 


-0. 072548 


-0.078011 


0.994438 


221 


« 


m 


0. 012463 




0.991223 


£21 


1 




-0. 082838 


-0. 087382 


0.993039 


221 


2 




0.035569 


0.035454 


0.999933 


£21 


2 


h 


0.024068 


0. 023727 


0.999973 


221 


2 


c 


0. 035314 


0. 035149 


0.999942 


£21 


2 


o 


0.040988 


0. 024147 


0. 987793 


221 


2 


• 


-0.000885 


-0.000885 


1.000000 


£21 


2 


f 


-0. 003799 


-0. 020276 


0. 989501 


221 


2 


g 


0. 035177 


0. 017615 


0. 987358 


£21 


2 


h 


-0. 034210 


-0. 040280 


0. 998956 


£21 




J 


-0. 066140 


-0. 076664 


0. 988966 


221 


2 


1 


-0. 001531 


-0. 001697 


0. 999999 




2 


m 


-0. 013498 


-0. 028967 


0. 987979 


221 


2 


• 


-0. 092854 


-0. 102614 


0. 990053 



Significance Level 



0.071199 
0. 404370 

0.000000 

0.498106 
0. 194354 
0. 333200 
0.144611 
0. 037527 
0. 173684 
0. 057249 
0. 403540 

0.000000 

0. 425594 
0.413433 
0. 497367 
0. 265752 
0. 524866 
0. 373397 
0.260147 



P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Slallarlty Measure: #1 « Cosine. #2 « Dice 
TM: Tera Weighting Scheaes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance judgaent and the 
systca's predicted relevance based on unresolved anaphors. rjf Is between 
the user's relevance judgaent and the systca's predicted relevance based 
on resolved anaphors. 

Because the user's judgaents were scaled froa low to high (1 « aost relevant, 
4 ■ aost non-relevant) a strong negative correlation shows agreeaent 
between user's and systea's relevance Judgaents. 

Significance Level: A positive Z indicates that the second correlation is higher 
than the first correlation (r4y.> rjy). If this Z is statistically 
significant as indicated by tne asterislcs, then resolving anaphors iaproves 
the lystM's predications of relevance. nQ<> 
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Page 

A Statistical Cowparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgaents: for Anaphoric Class 

NOMINAL SUBSTITUTES 



Correlation Coefficients Significance Level 



Q 


s 






ur 


z 










—0. 065303 


1 . 000000 


0.000000 






d 


4«7137 


-0. 399368 


0. 9927S5 


-0. 314716 






• 




-0. 000884 


1.000000 


0. 000000 


22£ 


1 


h 


-0. ld£4£l 


-0. 181549 


0. 999816 


-0. 206736 


£2£ 




J 




-0. 40£9£9 


0. 999621 


-0. 178444 


222 


1 


91 


-0. A3d5£4 


-0. 422595 


0.992656 


-0. 324801 


£-2£ 






-0. 194080 


-0. 173806 


0. 990370 


-0. 664064 


222 




a 


-0. 076£59 


-0. 076259 


1.000000 


0.000000 


222 


£ 


b 


-0. 0&3I33 


-0. 063133 


1.000000 


0. 000000 


222 


£ 


c 


-0> 0735S4 


-0. 073564 


1.000000 


0. 000000 


222 


£ 


d 


-0. 410846 


-0. 409847 


0. 99659£ 


-0. 059321 


222 


£ 




-0. 000884 


-0. 0008£i4 


1.000000 


0.000000 


222 


£ 




-0. 338840 


-0. 337874 


0. 995228 


-0. 046988 


222 


£ 


c 


-0.410581 


-0.409606 


0. 996027 


-0. 053649 


222 




h 


-0. 0££30d 


-0. 022249 


0. 999999 


-0. 223064 


222 


£ 


J 


-0. 430870 


-0. 430466 


0. 997729 


-0. 014989 


222 


£ 




-0. 00£454 


-0. 002454 


1.000000 


0. 000000 






m 


-0. 44£443 


-0. 441810 


0. 995928 


-0. 034932 


222 


£ 


n 


-0. £56865 


-0. 234907 


0. 987643 


-0. 643495 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Sinllarlty Measure: II « Coslr.s. 12 « Dice 
TU: Terai Weighting Schcaes: See Result Page R-1 

Correlation Coefficients: rj^ Is between the user's relevance judgnent and the 
systen's predicted relevance based on unresolved anaphors. Is between 
the user's relevance Judgnent and the systCM's predicted relevance based 
on resolved anaphors. 

Because the user's Judgnents were scaled froNi low to high (1 * nost relevant. 
4 ■ mst non-relevant) a strong negative correlation shows agreeMnt 
between user's and systen's relevance JudgMnts. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (r4r> rju)* this Z Is sUtlstlcally 
significant as Indicated by tne asterisks* then resolving anaphors Improves 
the ^fstM's predications of relevance. 
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t Appendix E-151 

Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



HOMIHAL SUBSTITUTES 



Correlation Coefficients Significance level 



Q 


S 


TM r. 






z 










J • 


''ur 










-0. 732594 








223 




d 


-0. £94873 


-0. 688107 


0. 996236 


-0. 400950 




•* 




-e.00dG4a 


-0. 000648 


1.000000 


0. 000000 






h 


-d. 37303& 


-0. 373034 


1. 000000 


-0. 333290 




1 


3 


-e. 4SS509 


-0.417174 


0. 999865 


-1.296866 


223 


1 


m 


-0. &9567& 


-0. 674388 


0. 997288 


-1.344104 






ri 


-01. £39829 


-0. 616235 


0. 997039 


-1.355332 




2 


a 


-0. 7541 3£ 


-0. 752364 


0. 999987 


-1. 630948 




^ 


t) 


-e. 718551 


-0. 717670 


0. 999996 


-1.532998 


2&3 


2 


c 


-e. 752835 


-0. 751203 


0. 999989 


-1.633879 


£23 


2 


d 


-0. 732325 


-0. 731320 


0.999396 


-0. 158490 


223 


2 


• 


-0.000648 


-0. 000648 


1.000000 


0. 000000 


223 


2 


f 


-0. 709220 


-0. 708283 


0. 999766 


-0.229166 


223 


2 


c 


-0.731842 


-0. 730878 


0. 999493 


-0. 165860 


223 


2 


h 


-0. SB115& 


-0. 361080 


1. 000000 


-0. 440584 


223 


2 


J 


-0. 389875 


-0. 387820 


0. 999958 


-0. 896217 


223 


2 


1 


-0. 315803 


-0. 315761 


1.000000 


-0. 647434 




2 


i« 


-0. 726S92 


-0. 719852 


0. 999597 


-1. 196002 




2 


n 


-0. 670259 


-0. 662359 


0. 999485 


-I. 154575 



P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlKFO 
S: Similarity Measure: II « Cosine. 12 « Dice 
TH: Tern Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rw Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 » most relevant, 
4 « most non-relevant) a strono negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation itwy rju). If this Z Is sUtlstlcally 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's ReVevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 

NOMINAL SUBSTITUTES 



Correlation Coefficients 



Significance Level 



Q 


s 


TM rj„ 


jr 


£27 


1 




-0. 128573 


-0. 128573 


£27 


1 


d 


-0. 321199 


-0. 351401 


227 




• 


-e. e00S8S 


-0. 000658 


227 


1 


h 


0. 333390 


0. 338163 


£27 


1 


J 


0. 0S9223 


0. 019964 


227 


1 


m 


-0. 227812 


-0. 253677 


227 


1 


n 


-0. 0£1523 


-0. 077847 


227 


2 


a 


-0. 12270S 


-0. 122705 


227 


2 


b 


-0. 133699 


-0. 133699 


227 


2 


c 


-0. 1 26353 


-0. 126353 


227 


2 


d 


-0. 288205 


-0. 285871 


227 


2 


« 


-0. 000658 


-0. 000658 


£27 


2 


f 


-0. 282278 


-0.255137 


227 


2 


5 


-0. 290104 


-0. 280970 


£27 


2 


h 


0. 331 1A8 


0. 336708 


£27 


2 


J 


0. 023736 


0. 000551 


227 


2 


1 


0. 037912 


0. 094723 


£27 


2 


jn 


-0.214012 


-0.211916 


227 


2 


n 


-0. 066552 


-0. 070422 





z 


ur 




1. 000000 


0. 000000 


0. 940715 


0.416855 


1.000000 


0. 000000 


0. 990107 


-0. 161140 


0. 945642 


0. 533209 


0. 927851 


0. 314029 


0. 928269 


0. 193250 


1.000000 


0. 000000 


1.000000 


0. 000000 


1.000000 


0. 000000 


0. 950694 


-0. 034733 


1.000000 


0. 000000 


0. 948825 


-0. 394053 


0. 947762 


-0. 132004 


0. 997306 


-0. 359012 


0. 954392 


0. 343405 


0. 985467 


-1.494170 


0. 939348 


-0. 027571 


0. 933953 


0. 047729 



P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 « Cosine. #2 » Dice 
TH: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rw Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 • most relevant, 
4 • most non-relevant) a strong negative correlation shows agreeaent 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation {T\r> ''ju)* ^^^s Z Is sUtlstlcally 
significant as Indicated by tne asterisks* then resolving anaphors Improves 
the liystam's predications of relevance. 
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Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judsments with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 



NOMINAL SUBSTITUTES 



Correlation Coefficients Significance Level 



Q 


S 


TW 




jr 


Mr 


z 




1 


A 


~0» 373570 


~0« 37.d3SS 


»• 999999 


—0. 96i;^605 




A 


0 


_A OIK 4 O^O 


~w« SSI See 


w* 73513b 


_A AA4 
~0. 0010^6 


23d 




• 


-0. 000600 


-0. 000600 


1.000000 


0. 000000 


£30 




h 


0. 07&7S9 


0. 077018 


0. 999997 


-0. 423936 


£30 




J 


0. 05S008 


0. 054187 


0. 996789 


0. 208135 


£30 




m 


-0. 102239 


-0. 1 12390 


0. 968626 


0. 177683 


£30 




n 


-0. 169797 


-0. 175705 


0. 967702 


0.102896 


£30 


2 




-0. 305620 


-0. 305273 


0. 999999 


-1. 108923 


£30 


2 


b 


-0. £34657 


-0. 234507 


1.000000 


-l.££e£04 


£30 


2 


c 


-0. 301274 


-0. 300957 


0. 999999 


-1. 117145 


£30 


2 


d 


-0. 197532 


-0. 207270 


0. 930252 


0. 116136 


£30 


2 


• 


>0. 000600 


-0. 000600 


1.000000 


0. 000000 


£30 


2 


f 


-0. 149564 


-0. 162809 


0.941879 


0. 171498 


i £30 


2 


B 


-0. 196159 


-0. 205976 


0. 925394 


0. 113172 


1 £30 


2 


h 


0. 064555 


0. 06491 1 


0. 999999 


-1. 178203 


£30 


2 


J 


0. 074391 


0. 069176 


0. 995794 


0. £4645£ 


i 230 


2 


▲ 


0. 073664 


0. 073596 


0. 999999 


0. 183££6 


■ £30 


2 


m 


-0. 075132 


-0. 089965 


0. 949348 


0. £03868 


£30 


2 




-0. 137167 


-0. 148929 


0. 945605 


0. 157109 



P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Sliillarlty Measure: II « Cosine. 12 ■ Dice 
TH: Tern Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 ■ most relevant. 
4 « most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rir> rju). If this Z Is statistically 
significant as Indicated by the asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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Appendix E-154 

Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors .and User's Relevance Judgments: for Anaphoric Class 



HOMIHAL SUBSTITUTES 



Correlation Coefficients Significance Level 



Q 


s 






''ur 


z 


235 


J 




-0. 031123 




X m WWW 




235 




d 


-0. 329375 






1. 7a7el9 


235 




• 


>0. 000931 


-0. 000931 


1.000000 


0. 000000 


£35 




h 


-3. 094045 


-0. 104226 


0.989185 


0. 295166 


£35 




J 


-0. 506407 


-0. 574452 


0.983211 


1.736605 


235 


i 


m 


-0. 397157 


-0. 467259 


0. 986296 


1.864332 


235 


1 


n 


-0. 457890 


-0. 525953 


0. 984831 


1.783855 


235 


2 




-0. 160684 


-0. 160684 


1.000000 


0. 000000 


£35 


2 


b 


-0. 226216 


-0.226216 


1.000000 


0. 000000 


235 


2 


c 


-0. 155708 


-0. 155708 


1.000000 


0. 000000 


£35 


2 


d 


-0. 361740 


-0. 420046 


0. 989110 


1.751795 


£35 


2 


• 


-0. 000931 


-0. 000931 


1.000000 


0. 000000 


£25 


2 


f 


-0. 412523 


-0. 475978 


0.984102 


1.615723 


235 


2 


S 


-0. 362648 


-0. 421563 


0. 988388 


1.717625 


£35 


£ 


h 


-0. 12&969 


-0. 142522 


0. 998556 


1.234739 


£35 


2 


J 


-0. 453489 


-0. 519651 


0. 983442 


1.671045 


235 


2 


A 


-0.014128 


-0. 015829 


0. 999985 


1. 316555 


£35 


a 


m 


-0. 403257 


-0. 467173 


0. 986074 


1.722661 


235 


2 


•n 


-0. 434261 


-0. 495595 


0. 985837 


1.661679 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 « Cosine. 12 « Dice 
TU: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance Judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's Judgments were scaled from low to high (1 » most relevant, 
4 • most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance Judgments. 

Significance Level : A positive Z indicates that the second correlation Is higher 
than the first correlation (r4r> rju)* If this Z Is sutlstlcally 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the lystm'f predications of relevance. 
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Appendix E-155 
Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments;: for Anaphoric Class 



NOMINAL SUBSTITUTES 



Correlation Coefficients Sloalflcance Level 





d 






ur 


7 


SA8 


4 

A 


a 


-0. 174181 


—0. 171062: 




— W. 99I2PS4S 


£48 


■« 


d 


-0. 3768£4 


-0. 440045 


• ^ w V# 




£48 




• 


-0.001000 


-0.001000 


1.000000 


0. 000000 


£48 


1 


h 


-0.209710 


-0. 209766 


1.000000 


0.926858 


£48 


I 


J 


-0. £40£05 


-0. 258084 


0. 998£40 


1.379506 


£48 




m 


-0.384S95 


-0. 429285 


0. 982199 


1. 132802 


£48 


4 


ri 


-0. 315971 


-0. 365961 


0. 985698 


1 . 384493 


£48 


£ 


a 


-0. 162555 


-0. 158754 


0. 999855 


-1.010935 


£48 




b 


-0. 1&2326 


-0. 160639 


0. 999969 


-0. 978255 


£48 


£ 


c 


-0. 185652 


-0. 162231 


0. 999882 


-1.006056 


£48 


£ 


d 


-0. 388644 


-0. 436862 


0. 957483 


0. 807662 


£48 


£ 


• 


-0.001000 


-0. 001000 


1.000000 


0. 000000 


£48 


£ 


f 


-0. 38781 1 


-0. 438841 


0. 953£13 


0.815199 


£48 


£ 


9 


-0.,38511£ 


-0. 432598 


0. 954699 


0. 769645 


£48 


£ 


h 


-0. 209574 


-0. 209517 


1 . 000000 


0.913745 


£43 


£ 


j 


-0. 253679 


-0.267125 


0. 998534 


1. 14£641 


£48 


£ 


i. 


-0. 206419 


-0. £06457 


1.000000 


0. 483405 


£43 


£ 


M 


-0. 4£&0&& 


-0. 454715 


0. 975104 


0. 636954 


£48 


£ 


n 


-0. 348954 


-0. 385543 


0. 982370 


0. 929850 



NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200>299 on PsychlNFO 
S: Similarity Measure: #1 ' Cosine. #2 « Dice 
TW: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 > most relevant. 
4 ■ most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first corrtlatlon (r4y.> rju). If this Z Is statistically 
significant as Indicated fay the asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 

NOMINAL SUBSTITUTES 



Q 


s 


TW 


r . 

ju 


S52 




a 


e. 087621 


252 




ti 


0. 14&321 


252 




• 


-0. 000931 


252 




h 


0.:187££1 


252 




J 


0. 209856 


252 




m 


0.200105 


252 




n 


0. 253456 


252 


2 


m 


0. 006594 


252 


2 


b 


-0. 029094 


252 


2 


c 


0. 003133 


252 


2 


d 


0.188045 


252 


2 


• ■ 


-0.000931 


252 


2 


f 


0. 095067 


252 


2 


S 


0. 196328 


252 


2 


h 


0. 024025 


252 


2 




0. 244992 


252 


2 


i 


0. 000861 


252 


2 


m 


0.248129 


252 


2 


• 


0. 295229 



Correlation Coefficients 



0. 087821 
0. 170472 

-0. 000931 
0. 192869 
0.243429 
0.224950 
0. 267481 
0. 006594 

-0. 029094 
0.009133 
0.204110 

-0. 000931 
0. 108899 
0.211898 
0. 025382 
0. 260262 
0. 000966 
0. 261814 
0. 301217 



ur 

1. 002)000 
0. 997623 

1.000000 

0. 9970S7 
0.991 lis 
0. 996364 
0. 996404 
1 . 000000 
1. 000000 

1.000000 

0. 996135 

1.000000 

0. 993730 
0. 995575 
0. 999952 
0.995688 
1.000000 
0. 995253 
0. 995104 



significance Level 



0. 000000 

-1.498251 

0. 000000 

-0. 294817 
-1 . 0921 12 
-1.257715 
-0. 724663 
0. 000000 
0. 000000 
0. 000000 
-0.789158 
0. 000000 
-0.526705 
-0.716091 
-0. 585217 

-0.719136 
-0.513527 
-0. 615062 
-0. 268839 



P > .05 



NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: II « Cosine. #2 « Dice 
TH: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance Judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 « most relevant, 
4 « most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation irw> rju). If this Z Is statistically 
significant as Indicated fay tne asterisks, then resolving anaphors Improves 
the vstw's predlcatlom of relevance. 
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Appendix E-1S7 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



PRO-VERBS 



Correlation Coefficients Stjnlfl^cance Level 



Q 


s 


TW r. 






z 












'^ur 






J 




-0. 105343 


-0. 106373 


0. 999971 


0. 590049 


203 




d 


-0. 141626 


-0. 140762 


0. 999953 


-0. 394116 


203 




• 


-0. 001022 


-0. 001022 


1.000000 


0. 000000 


203 




h 


0. 117064 


0. 1 17062 


1.000000 


-1.013372 


203 




J 


0. 072657 


0. 074912 


0. 999952 


-0.91913d 


203 




m 


O0. 997276 


-0. 095974 


0. 999947 


-0.555197 


203 




n 


-0. 012627 


-0.010621 


0. 999942 


-0. 606327 


203 


2 




-0. 042574 


-0. 043666 


0. 999949 


0. 565776 


203 


2 


b 


0. 005796 


0. 005309 


0. 999993 


0. 554977 


203 


2 


c 


-0. 037613 


-0. 036727 


0. 999963 


0. 566352 


203 


2 


d 


-0. 070321 


-0. 070567 


0. 999974 


0. 146746 


203 


2 


• 


-0. 001022 


-0.001022 


1.000000 


0. 000000 


203 


2 


f 


-0. 036263 


-0. 036419 


0. 999995 


0. 224600 


203 


2 


B 


-0. 070969 


-0. 071214 


0. 999979 


0. 153097 


203 




h 


0. 069005 


0. 069169 


1.000000 


-1.007216 


203 


2 


J 


0. 034705 


0. 035171 


0. 999979 


-0. 313427 


203 


2 


1 


0. 002329 


0. 002420 


1.000000 


-0. 9961 19 


203 


2 
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p > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: II * Cosine. #2 * Dice 
TM: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: r*^ Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance Judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's Judgments were scaled from low to high (1 » most relevant, 
4 • most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance Judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rj|.> rju). If this Z Is statistically 
significant as Indicated fay tne asterisks, then resolving anaphors Improves 
the system's predications of relevance. 

301 



Page 295 

, Appendix E-158 

Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



PRO-VERBS 



Correlation Coefficients Significance Level 



0 


s 


TH 


ju 


r . 


r 

ur 


2 


£07 


1 


a 


0. 089728 


0. 088733 


0. 999863 


0. 26990S 


207 


1 


d 


-0. 0e&0&4 


-0. 007045 


0. 999902 


0.31 £993 


207 






-0. 000958 


-0. 000958 


1.000000 


0. 000000 


207 


1 


h 


0. 213822 


0. £138£2 


1.000000 


0. 000000 


207 


1 


J 


0. 188212 


0. 188215 


1.000000 


-0. 016288 


207 


1 


m 


0.099118 


0. 099005 


0. 999977 


0. 075470 


207 


1 


n 


0. 084855 


0. 065020 


0. 999980 


-0. 1 15978 


207 


2 


a 


0. 089802 


0. 068528 


0. 999717 


0. £02447 


207 


2 


b 


0. 054739 


0. 054262 


0. 999938 


0. 188977 


£07 


2 


c 


0. 070283 


0. 069245 


0. 999769 


0.£1£20£ 


207 


£ 


d 


0.0S1932 


0. 022143 


0. 999743 


-0. 041548 


207 


2 


• 


-0. 000958 


-0. 000958 


1.000000 


0. 000000 


207 


2 


f 


-0. 040281 


-0. 041328 


0. 999955 


0. 491185 


207 


2 


g 


0.018805 


0. 01884£ 


0. 999799 


-0. 008286 


207 


£ 


h 


0. 202720 


0. £0£736 


1.000000 


-1.053751 


2«7 


£ 


J 


0. 193024 


0. 193329 


0. 999997 


-0. 597415 


207 


2 


2 


0. 175987 


0. 178003 


1.000000 


-0. 695023 


207 


2 


m 


0. 121848 


0. 1££559 


0. 999958 


-0. 35077A 


£07 


2 


* r% 


0. 098820 


0. 100521 


0. 999972 


-1.017328 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 « Cosine. #2 « Dice 
TU: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 * most relevant, 
4 « most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation {x\t> rju). If this Z 1i statistically 
significant as indicated by tne asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 

PRO-VERBS 



Correlation Coefficients 



Significance Level 
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p > .05 
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1.09£061 
1.339£04 
0. 305103 
0.813148 
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0. 33£593 
0. 738458 
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0. 000000 
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0.511488 



NOTES : 



Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 ■ Cosine. 12 « Dice 
TH: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the ystr*s relevance judgment and the system's predicted relevance based 



on resolved anaphors. 



Because the user's judgments were scaled from low to high (1 « most relevant, 
4 ■ most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rj|.> rju). If this Z Is sUtlstlcally 
significant as Indicated fey tne asterisks, then resolving anaphors Improves 
the lystam's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



PRO-VERBS 



Correlation Coefficients Significance Level 
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NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Neasure: #1 - Cosine. 12 - Dice 

» 

TU: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 « most relevant, 
4 ■ most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than tht first correlation Kt\t> rju). If this Z Is statistically 
significant as Indlcattd by tnt asterisks, then resolving anaphors Improves 
tht system's prtdlcatlont of rtlevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



PRO-VERBS 



Correlation Coefficients Significance Level 
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NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: II • Cosine. #2 « Dice 
TH: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's rtlevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 « most relevant, 
4 • most non-relevant) a strong negative correlation shows agreement 
between user's and system's rtlevance Judgments. 

Significance Level: A positive Z indicates that the second correlation Is higher 
than the first correlation (rir> rju). If this Z Is sUtlstlcally 
significant as Indicated by tM asterisks* then resolving anaphors Improves 
the nystem's predications ef rtlevance. 
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A Statistical Comparison of the Relationship Between 
* Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



PRO»YERBS 



Correlation Coefficients Significance Level 
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NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Simllarlty Neasun: 11 - Cosine. #2 « Dice 
TH: Term Weighting Schwis: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance judgment and the 
system's predicted rtletance based on unresolved anaphors. rjr Is between 
the user's relcvanct judgment and the system's predicted relevance based 
on resolved anaphors. 

Btcaust the user's judgments were scaled from low to high (1 • most relevant, 
4 • most non-relevant) a strono negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rjy.> rjy). If this Z Is statistically 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
tlM system's predlcatlent of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



PRO-VERBS 



Correlation Coefficients 
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P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: fl « Cosine. 12 • Dice 
TU: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance Judgment and the 
system's predicted reletance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 - most relevant, 
4 « most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than tht first correlation (rjr> rju). If this Z Is sUtlstlcally 
significant as Indicated bf tne asterisks, then resolving anaphors Improves 
tht system's predications of relevance. 
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A Statistical Conparlsbn of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 



PRO-YERBS 



Correlation Coefficients 



Significance level 
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P > .05 
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NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: SiMllaHty Neasure: #1 » Cosine. 12 » Dice 
TH: Teni Weighting Schnes: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance Judgwnt and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgnents were scaled fm low to high (1 • nost relevant, 
4 • nost non-relevant) a strong negative correlation shows ag ranw t 
between user's and systen's relevance judgnents. 

Significance Level: A posltl^ Z Indicates that the second correlation Is higher 
than the first correlation (rjr> rju). If this Z Is sutlstlcally 
significant as Indlated by tnt asterisks, then resolving anaphors Improves 
the lystea's predications of relevance. 
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Conparlson of the Relationship Between 

and User's Relevance Judgments with Resolved 

Relevance Judgnents: for Anaphoric Class 



PRO»yERBS 



Correlation Coefficients Significance Level 



y 


e 

d 


TW r. 






L 










•"jr 


•"ur 




248 


1 


a 


-0. 174181 


-0. 171062 


0. 999898 


-0. 990842 


248 


1 


d 


-0. 376824 


-0. 401778 


0. 995723 


1.285642 


248 


* 


• 


-0. 001000 


-0. 001000 


1.000000 


0. 000000 


248 


1 


h 


-0.209710 


-0.209715 


1. 000000 


1.908137 


248 


1 


J 


-0. 240205 


-0. 244108 


0. 999876 


1. 136981 


248 




m 


-0. 384995 


-0. 400380 


0. 998027 


1. 172069 


248 


1 


n 


-0. 315971 


-0. 330202 


0. 938450 


1. 194390 


248 


2 


a 


-0. 162555 


-0. 158754 


0. 999855 


-1.010935 


248 


2 


b 


-0. 162328 


-0. 160639 


0. 999969 


-0. 978255 


248 


2 


c 


-0. 165652 


-0. 162231 


0. 999882 


-1.006056 


248 


2 


d 


-0. 388644 


-0.410135 


0.997319 


1.398259 


248 


2 


• 


-0. 001000 


-0. 001000 


1. 000000 


0. 000000 


248 


2 


f 


-0.387811 


-0. 408083 


0. 998418 


1.696249 


248 


2 


S 


-0.385112 


-0. 407324 


e. 997400 


1.462456 


248 


2 


f» 


-0. 209574 


-0. 209576 


1.000000 


0. 653858 


1 248 


2 


J 


-0. 253679 


-0. 255578 


0. 999934 


0. 764880 


1 248 


2 


1 


-0. 206419 


-0. 206449 


1.000000 


1.938001 


248 


2 




-0. 426066 


-0. 440730 


0. 998495 


1.294952 


248 


2 


• n 


-0. 348954 


-0. 360402 


0. 998800 


1. 104669 



p > .05 



NOTES: 

Q: Queries 100-199 tiere searched on INSPEC: 200-299 on PsychlNFO 
S: SlallaHty Measure: #1 - COslne. 12 » Dice 
TM: Teni Welglitffig SchcMs: See Result Page R-1 

Corrtlatlon Coefficients: rju Is between the user's relevance JudgMnt and the 
systea's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgaent and the systca's predicted relevance based 
on resolved tnaphors. 

Because the user's Judgnents were scaled froa low to high (1 « nost relevant, 
4 ■ aost non-relevant) a strong negative correlation shows agreenent 
between user's end systea's relevance JudgMnts. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (r4r> rju). If this Z Is statistically 
significant is Indicated ly tne asterisks, then resolving anaphors Inprovcs 
the isrstea's predlcotlons of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 

PRO-VERBS 



Correlation Coefficients 
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NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 « Cosine. 12 « Dice 
TW: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rw Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgmtnU were scaled from low to high (1 « most relevant, 
4 « most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation {rw> rju). If this Z Is sutlstlcally 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the vstem's predications of relevance. 
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A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 



PRO-VERBS 



Correlation Coefficients Significance Level 
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NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: SlwllaHty Measure: #1 • Cosine. #2 - Dice 
TW: TcTfi Weighting ScheMs: See Result Page R-1 

Correlation Coefficients: rju is betMeen the user's relevance judgnent and the 
system's predicted relevance based on unresolved anaphors. rjr Is betneen 
the user's relevance judgwnt and the system's predicted relevance based 
on resolved anaphors. 

Because the user's Judgaents Mere scaled fron Iom to high (1 * mst relevant, 
4 ■ Mst non-relevant) • strong negative correlation shows agrewent 
between user's and systea's relevance JudgMnts. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rj|.> rju)- ^ ^* statistically 

significant as Indicated 1^ tne asterisks, then resolving anaphors Improves 
the lystOM's predications of relevance. ^ « « 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



PRO-VERBS 



Correlation Coefficients Significance Level 
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NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 « Cosine. #2 « Dice 
TH: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 « most relevant, 
4 • most non-relevant) a strong negative correlation shows agreement 
between user's and systemfs relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (r4r> rjy). If this Z Is statistically 
significant as Indicated fey tne asterisks, then resolving anaphors Improves 
the liystem's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



INDEFINITES 



Correlation Coefficients Significance Level 
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p > . 05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 « Cosine. 12 « Dice 
TU: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance Judgment and the system's predicted relevance based 
on r«!Solved. anaphors. 

Because the user's Judgments were scaled from low to high (1 « most relevant, 
4 • most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance Judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
thin the first correlation (r4r> rjy). If this Z Is statistically 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 
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Correlation Coefficients 
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NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 - Cosine. #2 - Dice 
TW: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance judgment and th« 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's prfidlcted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 > most relevant, 
4 ■ most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rjr> rju). If this Z Is statistically 
significant as Indicated by tnt asterisks, then resolving anaphors Improves 
the iiystiM's predications of relevance. 
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A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgaents with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 
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NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: SlaflaHty Ne^isurc: #1 >< Cosine. #2 « Dice 
TV: Tcm Nelghtliig ScbMs: Sec Result Page R-1 

CorrclatloR CDefflclcnts: rjy Is between the user's relevance Judgncnt and the 
sjrstca's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relcvmce JudgMnt and the sjrstca's predicted relevance based 
on resolved anepfcors. 

lecausc the user's Juitpw nts wire scaled fro« low to high (1 « Mst relevant. 
4 * Mst non-relevant) • stm^ negative correlation shows agreeMnt 
bet w t e ii user's and sjfstea's relevance Judgnents. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rjr> i^ju)* ^ ^> sutlstlcally 

slSnlflcant as ledlcatid bf the asterisks* then resolving anaphors Inproves 
the lofsteB's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 
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-0. 588325 


0. 996650 


-0. 331195 


219 


2 




-0. 616435 


-0. 616045 


0. 999783 


-0. 094833 


219 


& 


•fii 


-0. 504693 


-0. 498291 


0. 994031 


-0. £70583 


£19 


2 


ri 


-0.520418 


-0.512288 


0. 994977 


-0. 377661 



NOTES: 

Q: Queries 100>199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: #1 « Cosine. 12 » Dice 
TV: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance Judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's Judgments were scaled from low to high (1 « most relevant, 
4 « most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance Judgments. 

Significance Level: A positive Z indicates that the second correlation Is higher 
than the first correlation ir^y rjy). If this Z Is statistically 
significant as indicated bf urn asterisks, then resolving anaphors Improves 
the system's predicatioiis of rtlevenct. 



Page 310 

/ Appendix E-173 

Page 

A Statistical Coaparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 

INDEFINITES 



Correlation Coefficients 



Significance Level 



Q 


s 




* 


£21 




£2: 












2£1 


1 


£21 




£21 






X 


221 


2 


£21 




221 


2 


221 




221 


2 


£21 


2 


221 


2 


221 


2 


£21 


2 


221 


2 




2 






221 


2 



a 0 

C 0. 
• 

h -0. 

J -0, 

m 0, 

n -0. 

• 0. 

b 0. 

c 0. 

d 0. 

• -0. 
f -0. 
c 0. 
h -0. 

J -0. 

I -0. 

/n -0. 
n -0. 



044923 
075409 
000885 
034912 
072548 
012463 
082838 
035569 
024068 
035314 
040988 
000885 
e«3799 
035177 
034210 
066140 
001531 
013498 
092854 



0. 044756 
0. 052685 
-0. 000885 
-0. 03S029 
-0. 100S11 
-0. 022572 
-0. 097S57 
0. 035454 
0. 023727 
0. 035149 
0. 033995 
000885 
-0. 007429 
0. 028564 
-0. 034089 
-0. 069308 
-0. 001531 
-0. 015613 
-0.091717 



ur 



0. 999961 
0. 998553 

1 . 000000 
1.000000 

0. 997579 
0. 998193 
0. 999092 
0. 999933 
0. 999973 
0. 999942 
0. 997325 

1.000000 

0. 996078 
0. 996824 
1.000000 
0. 9994A6 

1 . 000000 

0. 998668 
0. 999901 



P > .05 



0. 071195 
1. 582524 

0. 000000 

1.220531 
:. 512321 
1. 556466 
1.305427 
0. 037527 
0. 173684 
0. 057249 
0. 357979 

0. 000000 

0. 153394 
0. 310647 
-2.221367 
0. 356998 

0. 000000 

0. 253694 
-0. 303504 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 

S: SiBllarlty Measure: #1 « Cdslne. 12 « Dice 

TW: Tera Weighting ScIwms: See Result Page R-1 * 

Correlation Coefficients: rju is between the user's relevance Judgment and the 
system's predicted reletance based on unresolved anaphors. rjr is between 
the user's relevance Judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's Judgments were scaled from low to high (1 » most relevant, 
4 ■ most non-relevant) a strong negative correlation shows ag re em en t 
between user's and system's relevance Judgmits. 

Significance Level: A positive Z indicates that the second correlation Is higher 
than the first correlation (rjr> >*ju)* 2 is sutistically 

significant as indicated ly the asterisks* then resolving anaphors improves 
the ^tem*s predications of relevance. 
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Page 

A Statistical Coaiparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 



INDEFINITES 



Correlation Coefficients Significance Level 





Q 


S 


TN 




• 


•"ur 


z 






1 


• 


-0. 96S303 


—0. 065303 


1.000000 


0. 000000 






1 


d 


407137 


-0. 335791 


0. 956934 


-1. 186072 




2ZZ 


• 


• 


-0. 000884 


-0. 000884 


1.000000 


0. 000000 




&2& 


1 


h 


-0. 182421 


-0. 185837 


0. 999604 


0. 572102 




SS2 




J 


-0. 403934 


-0. 371207 


0. 979887 


-0.787117 




222 


« 


HI 


-0. 430S£4 


-0.371317 


0. 971151 


-1. 186905 




222 


1 


n 


-0. 194080 


-0. 198344 


0. 992970 


0. 164022 




222 


2 


a 


-0. 07G259 


-0. 076259 


1.000000 


0.000000 




222 


2 


3 


-0. 083133 


-0.063:33 


1.000000 


0. 000000 




222 


2 


c 


-0. 073564 


-0. 073564 


1.000000 


0. 000000 




222 


2 


d 


-0. 410846 


-0. 375819 


0. 981931 


-0. 889177 




222 


2 


• 


-0. 000884 


-0. 000884 


1.000000 


0. 000000 




222 


2 


f 


-0. 338840 


-0.327114 


0. 989143 


-0. 377098 




222 


2 


5 


-0.410S81 


-0. 378757 


0. 982875 


-0. 831X85 


i 
1 


222 


2 


h 


-0. 022398 


-0. 023060 


0. 999994 


0. 982573 


222 


2 


J 


-0. 430670 


-0. 397667 


0. 980397 


-0. 813171 


X 


222 


2 




-0. 002454 


-0. 002538 


X. 000000 


0. 648074 




222 


2 


m 


-0. 442443 


-0. 407506 


0. 979663 


-0. 846871 




222 


2 




-0. 256865 


-0. 256679 


0. 990220 


-0. 006164 



NOTES: 

Q: Queries 100-ig9 were searched on INSPEC: 200-299 on PsychlNFO 
S: SlallaHty Heasure: #1 • Cosine. #2 • Dice 
TH: Tcni Weighting SchcMs: See Result Page R-1 

Conrclatfon Coefficients: rjy Is between the user's relevance judgnent and the 
systci's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance Judjwnt and the systcn's predicted relevance based 
on resolved anaphors. 

Because the user's Judgwnts were scaled froii low to high (1 • nost relevant, 
4 ■ nst non-relevant) a strong negative correlation shows agreenent 
between user's and systea's relevance JudgMnts. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rjr> rju)* this Z Is sUtlstlcally 
sffnif leant as Indicated by the asterisks, then resolving anaphors li^roves 
the lyttMi's predlcatloni of relevance. 
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Page 

A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 



INDEFINITES 



Correlation Coefficients 



Significance Level 



Q 


S 








1 




-0. 732594 


-0. 732594 


223 


« 


d 


-e. 694873 


-0. 678547 


223 






-8. 000648 


-0. 000648 


223 


4 

A. 


h 


-0. 373036 


-0. 372791 


223 


1 


J 


-0. 422509 


-0. 408334 


223 


1 


m 


-0.69S676 


-0. 644752 


223 


1 


n 


-0. 639829 


-0. £79432 


223 


2 


a 


-0. 754136 


~0. 754136 


223 


2 




-0. 718551 


-0. 718551 


223 


2 


c 


-0. 75283S 


-0. 752835 


223 


2 


d 


-0. 732325 


-0. 733488 


223 


2 




-0. 000648 


-0. 000648 


223 


2 




-0. 709220 


-0. 717798 


223 




9 


-0. 731842 


-0. 734684 


223 


2 


h 


-0. 361 158 


-0. 359903 


223 


2 


J 


-0. 389875 


-0. 384997 


223 


2 




-0. 315803 


-0.315320 


223 


2 


m 


-0. 726692 


-0. 704031 


223 


2 


r» 


-0. 670259 


-0. 639677 





7 


ur 




1.000000 


0. 000000 


0.960061 


-0.298708 


1.000000 


0. 000000 


0. 999999 


-0. 870524 


0. 999336 


-1.584656 


0.976911 


-1.126630 


0.974190 


-1.169080 


1.000000 


0. 000000 


1.000000 


0. 000000 


1.000000 


0. 00000^ 


0. 978226 


0. 030851 


1.000000 


0. 000000 


0. 984534 


0. 260585 


0. 979498 


0. »77675 


0. 999975 


-0. 702043 


0. 999839 


-1.083664 


0. 999997 


-0. 626624 


0.988673 


-0. 781386 


0. 986943 


-0. 903943 



P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 

S: Slallarlty Measure: #1 - Cosine. #2 - Dice 

TM: Terai Nelghting SdiCMs: See Result Page R-1 *' 

CDrrelatlon Coefficients: rjy is between the user's relevance Judgnent and the 
Siystea's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance Judgnent and the systcn's predicted relevance based 
on resolved anaphors. 

Because the user's Judgnents were scaled fro« low to high (1 • nost relevant, 
4 • nost non-relevant) a strong negative correlation shows agreenent 
between user's and systta's relevance judgnents. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (r4r> rju). If this Z Is statistical 1y 
significant as Indicated fey tie asterisks, then resolving anaphors Improves 
the lystci's predications of relevance. 
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Page 

A Statistical Coaparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgaents with Resolved 
Anaphors and User's Relevance Judgaents: for Anaphoric Class 

INDEFINITES 



Correlation Coefficients 



Significance Level 



TU 



227 
227 
227 
227 
227 
227 
227 
227 
227 
227 
227 
227 
227 
227 
227 
227 
227 
227 
227 



1 
1 
1 
1 
1 
1 
< 

2 
2 
2 
2 
2 

2 
2 
2 
2 
2 
2 



d 

h 

J 
m 
n 

b 
c 
d 
• 

f 

9 
h 

J 
m 

Ti 



-e. 128573 
-0.321199 
-0.000656 
0.333390 
0. 059223 
-0. 227612 
-0. 061523 
-0. 122705 
-0. 133699 
-0. 126353 
-0.268205 
-0.000658 
-0. 282278 
-0. 290104 
0.331148 
0. 023736 
0. 037912 
-0. 214012 
-0. 066552 



-0. 128573 
-0. 364801 
-0. 000658 
0.338044 
0. 037574 
-0. 268424 
-0.111317 
-0. 122705 
-0. 133699 
-0. 126353 
-0. 327784 
-0. 000658 
-0. 322813 
-0. 330734 
0. 336297 
0. 002447 
0. 09^589 
-0. 252393 
-0. 111433 



ur 

1.000000 

0. 949885 

1.000000 

0. 990107 
0. 968195 
0. 947537 

0. 960007 
1.000000 

1. 000000 
1.000000 
0.966682 
1.000000 
0.975858 
0. 967472 
0. 997306 
0. 97fl523 . 
0. 985453 
0. 964586 
0. 967363 



P > .05 



0. 000000 
0. 654837 
0. 000000 
-0. 157102 
0. 384391 
0. 578773 
0.790844 
0.000000 

0. 000000 

0. 000000 
0.719152 

0.000000 

0. 862224 
0. 747528 
-0.332534 

0.459486 
-1.489917 
0. 662769 
0. 789160 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Slallarlty Measure: #1 - Cosine. #2 « Dice 
TV: Tem Weighting SdiCMs: See Result Page R-1 

Cenrelatlon Coefficients: rju Is between the user's relevance Judgaent and the 
systM's predicted relevance htsed on unresolved anaphors. rjf> Is between 
the user's relevance judgasnt and the systca's predicted relevance based 
on resolved anaphors. 

Because the user's judgaents were scaled froa low to high (1 » aost relevant, 
4 » aost MNi-relevant) a strong negative correlation shows agr e ea en t 
between user's and systea's relevance judgaents. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation {rw> rju). If this Z Is sUtlstlcally 
significant as Indicated ^ tKe asterisks* then resolving anaphors laproves 
the lystci's predications of relevance. 
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A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgaents with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 

INDEFINITES 



Correlation Coefficients 



Significance Level 



V 


c 








7 

•> 


239 


1 


a 


-0. 373570 


-0. 373570 


1.000000 


0. 000000 


239 


1 




-0. 251948 


-0. 245952 


0. 973601 


-0. 117476 


239 


« 


• 


-0. 000600 


-0.00060O 


1.000000 


0. 000000 


239 


4 


h 


0. 076769 


0. 076757 


0.999979 


0.008106 


239 


• 

A. 




0.058008 


0. 065870 


0-997114 


-0. 451945 


239 


X 




-0. 102239 


-0. 094741 


0.985114 


-0. 190339 


239 


1 




-0. 169797 


-0.164029 


0.983825 


-0. 141787 


239 


2 


a 


-^.305620 


-0.305620 


1.000000 


0.000000 


239 


2 


b 


-0. 234657 


-0.234657 


1.000000 


0. 000000 


239 


2 


c 


-0. 301274 


-0. 301274 


1.000000 


0.000000 


239 


2 


d 


-0. 197532 


-0.200564 


0. 977652 


0. 063798 


239 


2 




-0.000600 


-0.000600 


1.000000 


0.000000 


239 


2 




-0. 149564 


-0. 156477 


0.963113 


0. 165920 


239 


2 


5 


-0. 196159 


~0. 199755 


0.976409 


0. 073618 


239 


2 


h 


0.064555 


0.068055 


0.999842 


-0. 861271 


239 


2 


w 


0. 074391 


0. 081242 


0. 997614 


-0. 433525 


239 


2 


* 


0.073664 


0. 074659 


0.999982 


-0. 722802 


239 


2 


m 


-0.075132 


-0. 077109 


0.982990 


0. 046880 


239 


2 


y. 


-0. 137167 


-0. 139718 


0. 980198 


0. 056424 



P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPCC: 200-299 on PsychlNFO 
S: SMIaHty Metsurt: II - COslnt. #2 * Dice 
TU: Tem Helghtfiig Schwei: Set Result Page R-1 

CorrelatloR Coefficients: rju Is bctuetn the user's relevance Judgment and the 
system's predicted rtletanct based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved amphors. 

Because the user's Judgments were scaled from lew to high (1 • most relevant, 
4 « most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance Judgments. 

Significance Uvtl: A positive Z Indicates that the second correlation Is higher 
then the first correlation (rjr> rju). If this Z Is sUtlstlcally 
significant as Indicated fey tne asterisks, then resolving anaphors Improves 
the item's predlcatlont of relevance. 
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Page 

A Statistical CoMparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgaents with Resolved 
Anapbors and User's Relevance Judgments : for Anaphoric Class 

* 

INDEFINITES 



Correlation Coefficients 



Significance Level 



H 


e 
5 




"jr 


^r 


z 


235 


1 


• 


-9. 031 123 






0. 000000 


235 


1 


d 


-0. 329375 


-0.323510 


0.998940 


-0. 570205 


S3S 


1 


• 


-0.090931 


-0.000931 


1.000000 


0.000000 


235 




h 


-0.0940^5 


-0. 069413 


0. 995133 


-1.062404 


235 


1 


J 


-0.506407 


59359^ 


0. 997486 


-0. 194924 


235 


1 


m 


-0. 397157 


-0.392297 


0. 998345 


-0. 389353 


235 


1 


n 


-0.457890 


-0.453587 


0. 997284 


-0. 277949 


235 


2 


a 


-0. 160684 


-0. 160684 


1.000000 


0.000000 


235 


2 


b 


-0.226216 


-0.226226 


1.000000 


0.000000 


235 


2 


e 


-0. 155706 


-0.155708 


1.000000 


0.000000 


235 


2 


d 


-0. 361740 


-0. 361747 


0.998940 


0.000722 


235 


2 


• 


-0.000931 


-0. 000931 


1.000000 


0.000000 


235 


2 


f 


-0. 412523 


-0-417331 


0. 998649 


0. 430351 


235 


2 


9 


-0.362648 


-0. 363351 


0.998869 


0. 067272 


235 


2 


h 


-0. 126989 


-0. 120016 


0.999415 


-0. 871020 


235 


2 


J 


-0.453489 


-0.455260 


0. 998212 


0. 140986 


235 


2 




-0. 014128 


-0.013326 


0.999993 


-0. 906030 


235 


2 


an 


-0.403257 


-0.403586 


0.998522 


0. 028018 


235 


2 


n 


-0.434261 


-0.434530 


0.997962 


0. 019860 



P > .05 



NOTES: 

Q: QiieHes 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: SMIaHty Measure: #1 - Cosine. #2 « Dice 
TM: Teni Melghtlng ScfcoKs: See Result Page R-1 

CBrrelatloii Coefffcfents: rjy Is between the user's relevance judgMnt and the 
SjftUm's predfcted relevance based on unresolved anaphors. rjr Is between 
tfee relevance judgMnt and the systca's predicted relevance based 
ee resolved anapbors* 



•teause the user's J y dg w nt s wire scaled frm low to b1<4i (1 * eost relevant, 
4 « eost non-relevant) e stroin negative correlation shows agreement 
b e tw e e n user's and sjfstea's relevance Judgeents. 

Significance Level: A positive Z indicates that the second correlation Is higher 
than tbe first correlatioe (rjr> rjy). If this Z Is sUtlstically 
sfgeificaet as indicated ly tie asterisks, then resolving anaphors leproves 
tbe systoB's prodicatioet of relevance. 
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Page 

A Statistical COMparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgaents with Resolved 
Anaphors and User's Relevance Judgaents: for Anaphoric Class 



INDEFINITES 



Correlation Coefficients Significance Level 



Q 


c 








z 




« 

A. 




-0. 174181 


-0. 171062 


0. 999838 


-0. 330842 




1 


d 


-0. 378824 


-0. 401778 


0. 335723 


1.285642 


246 




« 


-0.001000 


-0. 001000 


1.000000 


0.000000 


24S 


1 


h 


-0. 209710 


-0. 209715 


1.000000 


1.308137 


248 


* 


J 


-0.240205 


-0. 244108 


0.393876 


1. 136381 


248 




m 


-0.384935 


-0.400380 


0. 338027 


t. 172063 


248 


1 


Tt 


-0. 315971 


-0.330202 


0.998450 


1. 134330 


248 


2 


a 


-0.162555 


-0. 158754 


0.933855 


-1.010335 


248 


2 


b 


-0. 162328 


-0. 160639 


0.339969 


-0. 378255 


248 


2 


c 


-0. 165652 


-0. 162231 


0.993882 


-1.006056 


248 


2 


d 


-0.388644 


-0.410135 


0. 397313 


1.338253 


248 


2 




-0.001000 


-0.001000 


• WW WW 


0.000000 


248 


2 


• 


-0.387811 


-0.408083 


0.398418 


1.636249 


248 


£ 


s 


-0.385112 


-0.407324 


0.997400 


1.462456 


248 


2 


H 


-0.209574 


-0.209576 


1.000000 


0. 653858 


248 


2 


J 


-0. 253679 


-0. 255578 


0.999934 


0. 764880 


248 


2 


« 


-0. 206419 


-0.206443 


1.000000 


1.938001 


248 


2 


HI 


-0. 426066 


-0. 440730 


0. 998495 


1.234352 


248 


2 


n 


-0. 348954 


-0.360402 


0. 998800 


1. 104663 



NOTES: 

Q: QueHcs 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: SlMflarfty Keasure: #1 * Cosine. #2 * Dice 
T»: Teni Helghtlng SchcMs: Sec Result Page R-1 

Carrelatfon Gaefflcfents: rj^ Is between the user's relevance Judgaent and the 
systM's predicted relevance tased on unresolved anaphors. rjr Is between 
the user's relevance Judgacnt and the sjrstca's predicted relevance based 
on resolved anaphors. 

•ecause the user's Judgaents were scaled froa low to high (1 • aost relevant, 
4 ■ aost non-relevant) • strong negative correlation shows agrceacnt 
bet w ew user's and sjrstea's relevance Judgaents. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation ir*r> rjy). If this Z Is sUtlstlcally 
ffgaff leant as Indicated ly tne asterisks, then resolving anaphors laproves 
the t9%tm'% predlcatloas of rtlevMce. 
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Page 

A Statistical Coaparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 
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NOTES: 

Q: Queries 100-19g were searched on IKSPEC: 20G-29g on PsychlNFO 
S: SMIaHty Iteasiirc: 11 « Cosine. #2 « Dice 
TV: Teni Weighting ScfccMs: See Result Page R-1 

CorrelatlOii Coefficients: rjy is be twe en the user's relevance Judgocnt and the 
systea's predicted relevance tesed on unresolved anaphors. rjr 1< between 
the user's relevance judgMRt and the systca's predicted relevance based 
on resolved anaphors. 

Recauso the user's Judgoants were scaled froa low to high (1 « aost relevant, 
4 • oost non-relevant) a strong negative correlation shows agrecacnt 
betwaen user's and sjfstea's relevance judgaants. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
then the first correlation ir*r> rju)- this Z Is sUtlstlcally 
significant as Indlcatod tf tie asterisks, then resolving anaphors liproves 
the Kystaa's predications of relevance. 
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/ Appendix E-181 

Page 

A Statistical Coaparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



RESIPUAt ADJECTIVES 



Correlation Coefficients Significance Level 
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NOTES: 

Q: Queries 100>199 were searched on INSPEC: 200-299 on P^hlNFO 
S: SfallaHty Measure: #1 « Cosine. iZ « Dice 
TH: Ttni Weighting SchcMs: See Result Page R-1 

Csrrelatloii Coefficients: rjy fs bctneen the user's relevance judgntnt and the 
systM's predicted reletaoct based on unresolved anaphors. rw Is between 
the user's rtlcvanct judgwit and the sjrsteK's predicted relevance based, 
on resolved anaphors. 

lecause the user's judgaents were scaled frM lew to high (1 • nost relevant, 
4 • aost non-relevant) e strong negative correlation shows agrecncnt 
between user's and sjfstea's relevance judgnents. 

Signlflcenc* Level: A posftfvt Z Indicates that the second correlation Is higher 
then the first correlatfoii (rjr> rju). If this Z Is smistically 
s1«nff leant as Indicated kf the asterisks, then resolving anaphors Inproves 
the iQfStea's predications of relevance. 
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r Appendix E-182 

Page 

A Statistical Coiiparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgaents: for Anaphoric Class 



RESIDUAL ADJECTIVES 



Correlation Coefficients Significance Level 

5 -^ju "^jr ^r ^ P > 
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NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNTO 
S: Siaflarity Measure: 11 « Cosine. f 2 » Dice 
TH: Teni Heiglitiiig SchcMs: See Result Page R-1 

CorrtlatlON CDtfflcieiits: rjy is between the user's relevance judgu e nt and the 
^ystfli's predicted relevance based on unresolved anaphors. rjr is between 
the user's rtlevanct judgnent and the systcn's predicted relevance based 
on resolved aiapliors. 

lecauso tbe user's Judgnents were scaled fren low to high (1 • nost relevant. 
4 » nost non-relevant) a strong negative correlation shows agrecncnt 
botMon user's and systaa's relcvence judjwnts. 

Significance Level: A positive Z Indlcatos that the second correlation is higher 
than the first correlation (rjr> rjy). If this Z is statistically 
significant as Indicated fey toe asterisks* then resolving anaphors inproves 
tbe ^tci's prodlcotlont of relevance. 
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Page 

A Statistical Co»par1son of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgments : for Anaphoric Class 



RESIDUAL ADJECTIVES 



Correlation Coefficients 



Significance Level 



Q 


S 




jr 


r 

ur 


z 


P > 


Sis 


1 


a 


-9, 532393 


-0. 534533 


0. 999965 


:. 423616 




212 


1 


d 


-9. £39979 


-0. 617442 


0.995464 


-0. 875175 




. 212 


1 


• 


-0.261644 


-0. 311513 


0. 962702 


0. 62136X 




212 


1 


h 


e. 669775 


0. 089747 


1.000000 


1.676563 




212 


♦ 


J 


-6.592539 


-0.580117 


0. 996792 


-0. 921043 




212 


I 


m 


-6. 663788 


-0.650951 


0.996615 


-0. 987932 




212 




n 


-6. 701485 


-0. 686023 


0. 997651 


-1.437058 




212 


2 


• 


-0. 553674 


-0.556X93 


0.999956 


1.510422 




212 


2 




-0.536116 


-0. 537247 


0.999990 


1.407011 




212 


2 


c 


-0.560076 


-0.562276 


0. 999967 


1.525915 




212 




d 


-0.683127 


-0. 684623 


0. 997974 


0. :S7717 




212 


2 




-0.266411 


-0.299793 


0. 988173 


0. 379156 




212 


2 


f 


-0.681729 


-0.687163 


0.996602 


0. S81362 




212 


2 


5 


-0.692679 


-0.694815 


0.998031 


0.231210 




212 


2 


h 


0.055266 


0.055208 


1. 000000 


2. 446004 




212 


£ 


J 


-0.685346 


-0.685642 


0.999939 


0. 045355 




212 


2 


> 


0.004343 


0. 004343 


1.00000e> 


0.000001? 




212 




m 


-0. 710908 


-0.711490 


0. 998727 


0. 080444 




212 


2 


n 


-0.731150 


-0. 726661 


0. 999566 


-0.513691 





05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: SlallaHty Neasure: «1 • Cosine. #2 » Dice 
TW: TeraHelghtlng SdMMs: See Result Page R-1 

Csrrelatlon Coefficients: rjg Is betMeen the user's relevance Judgnent and the 
systOM's predicted relevance based on unresolved anaphors. rw Is betueen 
tke user's relcvanct judgMnt and the systoi's predicted relevance based 
on resolved antphors. 

Recaust tbe user's Jiidjmfs uerc scaled frai lew to high (1 « nost relevant, 
4 • wnX non-relevent) t strong negative correlation shows a gr e w wn t 
be tw een user's and system's relevance judgnents. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
tban tbe first correlation (rir> rjy). If this Z Is sutlstlcally 
significant as Indicated br tne asterisks, then resolving anaphors Inproves 
tbe systfli's prtdlcatloni of relevance. 
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Page 

A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 



RESIDUAL ADJECTIVES 



Correlation Coefficients 
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NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Slullarlty Nttsurt: #1 » Cosine. #2 * Dice 
TV: Tern Weighting Srii— s; Sec Result Page R»l 

Correlation Coefficients: rjy Is between the user's relevance judgaent and the 
systca's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance Judjmnt and the systCM*s predicted relevance biased 
on resolved antphors. 

lecause the user's Judgwnts were scaled frai low to high (1 * aost relevant, 
4 • aost non-rtlevant) • strong negative correlation shows agrecncnt 
between user's and syste«*s relevance Judgwnts. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation itj^y rjy). If this Z Is sUtlstlcally 
sfgiilf leant as Indicated ly the asterisks, then resolving anaphors Inproves 
the iQfttan's predications of relevance. 
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, Appendix E-185 

Page 

A Stttlstical Coaptrlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgaents: for Anaphoric Class 

RESIDUAK ADJECTIVES 



Correlation Coefficients Significance Level 
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P > .05 



NOTES: 

Q: Queries 100>199 were searched on INSPEC: 200-299 on PsychlNFO 
S: SiBllarlty Nt^surt: II > Cosine. «2 » Dice 
TV: Tern Helghtlnj SdMMs: See Result Page R-1 

Correlation Coefflclanto: rju between the user's relevance Judgiwit and the 
systaa's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgMnt and the systoi's predicted relavance based 
on resolved anaphors. 

lecause the user's JudgMots were scaled froa low to high (1 • aost relevant, 
4 > »st non-relevant) a strono negative correlation shows agrecacnt 
between user's and systea's relevance judgaants. 

Signlflcinct Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation irwy rj^). If this Z Is sUtlstlcally 
tifnif leant as Indicated fey the asterisks • then resolving amphors Inproves 
the «yst«i*s predications of relevance. 
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A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 



RESIDUAL ADJECTIVES 



Correlation Coefficients Significance Level 
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p > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Sivtlarlty Measure: #1 » Cosine. 12 - Dice 

» 

TU: Teni Weighting SchCKs: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance judgment and the 
systea's predicted romance based on unresolved anaphors. rjr Is between 
the user's relevance Judgwnt and the systm's predicted relevance based 
on resolved anaphors. 

Because the user's judgMnts were scaled fron low to high (1 * most relevant, 
4 • Mst non-relevant) a strone negative correlation shows agremtnt 
betwten user's and systea's relevance judgMnts. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rj,.> rju). If this Z is sUtlstlcally 
significant as Indicated fey tlie asterisks, then resolving amphors Improves 
the lystei's predications of relevance. 
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Page 

A Statistical Coiiparlson of the Relationship Between 
Unfesolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Re1r>*ance Judgaents: for Anaphoric Class 



RESIDUAL ADJECTIVES 



Correlation Coefficients 



Significance Level 
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-8. 152344 


223 


•* 


n 


-8. 639829 


-8. 639824 


8. 999351 


-8. 188592 


223 


2 


« 


-8.754136 


-8.754136 


1.888888 


8. 888880 


223 


2 


b 


-8. 718551 


-8. 718551 


1.888888 


8. 888888 


223 


2 


c 


-8. 75283S 


-8. 752835 


1.888888 


8. 888888 


223 




a 


-8. 732325 


-8. 731614 


8.999758 


-8. 177229 


223 


2 


9 


-8. 888648 


-8. 888648 


1.888888 


8^ 888888 


223 




f 


-8. 789228 


-8. 788881 


8. 999S24 


-8. 235868 


223 


2 


5 


-8. 731842 


-8. 738932 


8. 999726 


-8.2:2666 


223 


2 


h 


-8. 361 158 


-8.361152 


1.888888 


-8. 32S827 


223 


2 


J 


-8. 389875 


-8. 369766 


8. 999997 


-8. 163723 


223 


2 




-8. 315683 


-8. 315883 


1.888888 


8. 88888«^ 


223 


2 


.m 


-8. 726692 


-8. 725628 


8. 999753 


-8.211118 


223 


2 


n 


-8. 678259 


-8. 668484 


8.999639 


-8. 331834 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Slallarlty Measure: II « Cosine. 12 • Dice 
TW: Teni Weighting SchcMS: See Result Page R-1 

Correlation Coefficients: r«u Is between the user's relevance Judgnent and the 
syst<M*s predicted relevance based on unresolved anaphors. rjr Is between 
the ustr's relevance Ju dgwn t and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgnents were scaled froM low to high (1 • nest relevant, 
4 ■ iBSt non-relevant) a strong negative correlation shows agreeMnt 
between user's and systOM's relevance judgMnts. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rjr> rju). If this Z Is sutlstlcally 
significant as Indicated tgr the asterisks, then resolving anaphors Improves 
the lyttaa's predications of relevance, 
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A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



RESIDUAL ADJECTIVES 



Correlation Coefficients Significance Level 



0 


s 




r . 


r 

ur 


2 


227 


1 




-0. 128573 


-9. 128573 


1.999999 


9. 999999 


227 




d 


-e. 321199 


-9. 376555 


9. 982496 


1. 385995 


227 


1 


• 


-0.00O6S8 


-9.999658 


1.999999 


9. 999999 


227 


t 


ft 


9. 333390 


•.337958 


9991 19 


-9. 154236 


227 


1 


J 


0. 959223 


9.928478 


9.993779 


1.234947 


£27 


1 


m 


-9.227812 


-9.282646 


9.981532 


I. 319515 


227 


2 


n 


-9. 981523 


-9. 111859 


9. 982482 


1.297144 


227 


2 


« 


-9. 12279S 


-9. 122795 


1.999999 


9, 999999 


227 


2 


b 


-9. 133699 


-9. 133699 


1.999999 


9. 999999 


227 


2 


c 


-9. 126353 


-9. 126353 


1.999999 


9. 990999 


227 


2 


d 


-9. 288295 


-9. 349998 


9. 984912 


1.419279 


227 


2 


• 


-9.999658 


-9.999658 


1.999999 


9. 999999 


227 


2 




-9. 282278 


-9. 331979 


9. 984462 


1.319522 


227 


2 


5 


-9. 299194 


-9. 344191 


9. 984239 


1.414211 


£27 


2 


h 


9. 331 148 


9. 336497 


9. 997319 


-9. 339819 


227 


2 


J 


9. 923736 


-9.999689 


9. 992821 


1.248916 


227 


2 


I 


9.937912 


9. 994589 


9. 985453 


-1.489917 


227 


2 




-9. 214912 


-9. 265246 


9. 984349 


1.325427 


227 


2 


n 


-9. 966552 


-9. 1 15933 


9.964419 


1.233237 



P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: S1«t1ar1ty Heasure: #1 • Cosine. «2 « Dice 
TW : Teni Helghtlng ScheMs: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance Judgpent and the 

Sstea's predicted relevance based on unresolved anaphors. rw Is between 
e user's rtlevanct Judgwnt and the systen's predicted relevance based 
on resolved anaphors. 

Beciust the user's JudgMnts were scaled froa low to high (1 « nost relevant, 
4 " wst nen-reltvant) a strong negative correlation shows agrecnent 
b et man user's and systea's relevance Judgnents. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first cerrtlatlon (rjr> rju). If this Z Is sUtlstlcally - 
sigaff leant at Indicated ly tne asterisks, then resolving anaphors Improves 
tbe tsf%tm*g fndicatloni 9f relevance. 
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A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgaents with Resolved 
Anaphors and User's Relevance Judgaents: for Anaphoric Class 



RESIDUAL ADJECTIVES 



Correlation Coefficients Significance level 



^ 5 ^Su '^jr ^r ^ P > 



230 1 a -0.373570 -0. 373S70 1.000000 

230 1 d -0.251948 -0.228853 0.953384 

230 1 • -0.000600 -0.000600 1.000000 

230 1 h 0.076769 0.102748 0.995429 

230 1 J 0.058008 0.071562 0.996303 

230 1 m -0. 102239 -0. 079787 0.975782 

230 1 n -0.169797 -0.140125 0.973112 

230 2 a -0.305620 -0.305620 1.000000 

230 2 b -0.234657 -0.234657 1.0O0000 

230 2 c -0.301274 -0.301274 l.OOOOOO 

230 2 d -0.197532 -0.196111 0.961831 

230 2 • -0.000600 -0.000600 l.OOOOOO 

230 2 f -O. 149564 -O. 154653 0. 970621 

230 2 9 -0- 196159 -O. 196062 0.959678 

230 2 h 0.064555 0.083648 0.997631 

230 £ J 0.074391 0.075794 0.997745 

230 2 I 0.073664 0.080383 0.99968^ 

230 2 4ti -0.075132 -0.075789 0.972691 

230 2 n -0.137167 -0. 134730 0.969785 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: SiMllarlty Measure: #1 - Cosine. 12 « Dice 
TV: Tera Weighting Scheaes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance judgaent and the 
systM's predicted relevance based on unresolved anaphors. rjr> Is between 
tht user's relevance judgaent and the systca's predicted relevance based 
on resolved anaphors. 

Because the user's Judgaents were scaled froa low to high (1 • aost relevant, 
4 ■ aost non-relevant) a strona negative correlation shows ag re went 
betwten user's and systea's relevance Judgaents. 

Significance Level: A positive Z IndlcaUs that the second correlation Is higher 
than the first correlation (rjr> rjy). If this Z Is sutlstlcally 
significant as Indicated bff tne asterisks, then resolving anaphors laproves 
the 4fst«i*s predications of relevance. 

333 



0. 000000 
-0. 339792 

0. oooooo 

-1. 188443 

-0. 688479 
-0. 446583 
-O. 564495 

O. OOOOOO 
, 0. OOOOOO 

0.000000 
-O. 022878 

O. OOOOOO 

O. 092608 
-O. 001521 
-1.211830 

-0.091302 
-1. 168519 
0.012299 
-O. 043617 
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k Statistical Coaparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 

RESIDUAL ADJECTIVES 



Correlation Coefficients 



Significance Level 



Q 


s 


235 


1 


235 


1 


S3S 


1 


235 


1 


235 


1 


235 


1 


235 


« 


235 


2 


235 


2 


235 


2 


235 


2 


235 


2 


235 


2 


235 


2 


235 


2 


235 


2 


235 


2 


235 


2 


235 


2 



TH 



« -9.931123 
d -0.329375 

• -9.990931 
h -9.994045 

J -9.506407 
«• -0.397157 
n -0.457890 
a -0.160684 
b -O. 226216 
c -0.155708 
d -0.361740 

• -0.000931 
f -0.412523 
9 -0.362648 
h -0.126989 

J -0.453489 
I -0.014128 
m -0.403257 
n -O. 434261 



-0. 031 123 
-O. 328414 
-9.0OO931 
-O. 094037 

-0.504925 
-0. 395828 
-O. 456009 
-0. 160684 
-O. 226216 
-O. 155708 
-O. 360974 
-0.000931 
-0.412119 
-0.361956 
-O. 126878 

-O. 452531 
-O. 014068 
-O. 402397 
-O. 433189 



ur 

X • 000090 
0.999977 

1.1 
1.1 

0.999937 
0.999969 
0.999933 
1.9099O9 
1.909909 
1.999909 
9.999981 
1.0999O9 
9.999994 
9. 999984 
1. 900009 

9. 999976 
1.999909 
9.999981 
9. 999975 



P > .05 



9.999999 
-9. 639002 

o.oooooe 

-9. 671878 
-9. 644994 
-9.661998 
-9. 766979 
9.999999 
9.999999 
9.O9990O 
-9. 563587 
9.999999 
-9. 529427 
-9. 5S9762 
-9. 649796 

-9. 659695 
-9. 641533 
-9. 634816 
-9.713149 



NOTES: 

Q: Queries 100-199 tiere searched on INSPEC: 200-299 on PsychlNFO 
S: Slollarlty Measure: #1 « Cosine. #2 - Dice 
TH: Tens Weighting SchtMs: See Result Page R-1 

Correlation Coefficients: rjy Is betMoen the user's relevance iudg^nt and the 
syste»*s predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgatnt and the systco's predicted relevance based 
on resolved anaphors. 

Because the user's judgMnts were scaled froa low to high (1 - eost relevant. 
4 ■ Host non-relevant) a strong negative correlation shows agrecmmt 
be tw e en user's and sjfstOM's relevance JudgRtnts. 

SIgnlflcanct Level: A positive Z Indicates that the second correlation Is higher 
tliin the first correlation (rjr> rju). If this Z Is statistically 
tfgnlf leant as Indicated ly the asterisks* then resolving amphors iMproves 
the «ysteM*t predications of relevance. 
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A Statistical Coaparlson of the Relationship Betneen 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 

RESIDUAL ADJECTIVES 



Correlation Coefficients 



Significance Level 



Q 


s 




r . 


r 

ur 


2 


248 


1 


* 


-0. 174181 


-0. 171062 


0.999898 


—0. 990842 


248 


1 


d 


-0. 376824 


-0.432920 


0. 961978 


0. 986158 


248 


1 


• 


-0. 001000 


-0.001000 


\m 900000 


0.000000 


248 


1 


h 


-0. 209710 


-0.209635 


0.999999 


-0.249116 


248 


« 


J 


~0. 24020S 


-0. 243935 


0.999592 


0. 601316 


248 


• 


m 


-0.38499S 


-0.378511 


0.985321 


-0. 163279 


248 


1 


n 


-0. 315971 


-0. 320346 


0.991-109 


0. 154769 


248 


2 


« 


-0. 162555 


-0. 158754 


0.9998S5 


-I. 010935 


248 


2 


b 


-0. 162328 


-0. 160639 


0.999969 


-0. 978255 


248 


2 


c 


-0. 165652 


-0. 162231 


0.999882 


-1.006056 


248 


2 


d 


-0.388644 


-0.458035 


0.950784 


1. 079765 


248 


2 


• 


-0.001000 


-0.001000 


1.000000 


0.000000 


248 


2 


f 


-0.387811 


-0. 464474 


0.932274 


1.021533 


248 


2 


8 


-0. 3851 12 


-0. 457215 


0.945090 


1.062198 


248 


2 


h 


-0.209574 


-0.209490 


0.999996 


-0. 137185 


248 


2 


J 


-0. 253679 


-0. 246109 


0.998955 


-0. 752547 


248 


2 


I 


-0. 206419 


-0. 208055 


0. 998351 


O. 130231 


248 


2 


m 

• 


-0. 426066 


-0. 402240 


0. 964065 


-0. 436521 


248 


2' 


n 


-0. 348954 


-0. 337148 


0. 974595 


-O. 249442 



P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Slnilarity Neasure: 11 • Cosine. #2 ^ Dice 
TU: Tem Weighting ScheMs: See Result Page R-1 

Carrelatlon Coefficients: rj^ Is between the user's relevance judgaent and the 
system's predicted relevance based on unresolved anaphors. rjr is between 
the user's relevance Judjwnt and the systci's predicted relevance based 
on resolved anaphors. 

Because the user's judgwots were scaled fro* low to high (1 • wost relevant, 
4 • «ost non-relevant) a strong negative correlation sh9ws agre«Mnt 
between user's and systea's relevance Judgwnts. 

Significance Level: A positive Z indicates that the second correlation Is higher 
than the first correlation {,tw> rjy). If this Z is sutistically 
significant as indicated fcy tne asterisks, then resolving amphors inproves 
the «yst«i's predications of relevance. 
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A Stitlstlcil Coaparfsbn of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgaents: for Anaphoric Class 

RESIDUAL ADJECTIVES 



Correlation Coefficients 



Significance Level 



252 
252 
252 
252 
252 
252 
252 
252 
252 
252 
252 
252 
252 
252 
252 
252 
• 252 
I 252 
252 



I 
1 



1 
1 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 



TU 

• o. 

d 0. 

h 0. 

J 0. 

m 0. 

n 0. 

a 0. 

b -0. 

c 0. 

d 0. 

• -0. 
f 0. 
8 0. 
h 0. 

J 

I 0. 

m 0. 

•n 0. 



087821 

148321 

000931 

187661 

203856 

200105 

253456 

006594 

#29094 

009133 

188045 

000931 

095067 

196328 

024025 

244992 

000861 

248129 

295229 



0. 087821 
0. 142479 
090931 
0.192027 
0.220785 
0.198632 
0.252892 
0.006594 



0.009133 
0. 182595 
-0.000931 
0.089059 
O. 190393 
O. 024478 
0.246097 
O.O0O696 
0. 244324 
0.294767 



ur 

1.OOOO00 
0.999663 
1.OOOOO0 
0.998954 
O. 995109 
0.999614 
0.999964 
1. 
1. 

1. 000000 
0.999405 
1.0O0OOO 
0.999268 
0.999344 
0.999987 
0.999414 
1.000000 
0.999630 
0.999967 



P > .05 



O.O0OOOO 
0.633625 
0.OOO00O 
-0.412223 
-0.479707 
0.229734 
O. 292642 



0.000000 
0.681319 
0.000000 
O. 668736 
O. 707253 
-O. 375015 
-O. 141210 
-O. 374171 
O. 514756 
0.25O8O4 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Slallarlty Measure: #1 • Cosine. #2 « Dice 
TU: Teni Uelghting SchtMs: See Result Page R-1 

Correlation Coefficients: rjy fs between the user's relevance Judgaent and the 
systea's predicted relevance " — " ' " — " — - " ' 



tesed on unresolved anaphors. rw Is betwien 
; and the systea's predicted relevance based 



the user's relevance Judgaent 
on resolved anaphors. 

•ecause the user's judgaents were scaled froa low to high (1 » aost relevant, 
4 • aost non-rtlevant) a strono negative correlation shows agrecacnt 
botwten user's and systea's relevance Judg w nts. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation {u^y rjy). If this Z Is sUtlstlcally 
significant as Indicated ly the asterisks, than resolving anaphors laproves 
the «ystai's prodlcatloNi of rolevance. 
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A Statistical Coaparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgsents with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



APyERBS 



Correlation Coefficients Significance Level 



Q 


s 


TM r. 




r 


z 












ur 








* 


-0. 105343 


-O. 106148 


0.999980 


0. 553864 


203 


• 


d 


141626 


-O. 142082 


0. 999987 


0. 396757 


203 


1 


• 


-0. 001022 


-0. 091022 


l.OOOOOO 


o.oooooo 


203 


1 


h 


0. 117064 


0. 1 17064 


1.000000 


o.oooooo 


203 


I 


J 


0. 072SS7 


0.072786 


l.OOOOOO 


0.411022 


203 


« 


m 


-0. 097276 


-0.097666 


0.999994 


0. 513715 


203 


I 


n 


-0. 012827 


-0. 013044 


0.999999 


0. 849521 


203 


2 


a 


-0. 042574 


-O. 043710 


O. 999964 


0. 582901 


203 


2 


:» 


0. 005796 


0. 00S370 


0.999995 


0. 574551 


203 


2 


c 


-0. 037613 


-0.038579 


0.999974 


0.585132 


203 


2 


d 


-0. 070321 


-0.070961 


0.999984 


0. 501539 


203 


2 


• 


-0. 001022 


-0.0O1022 


l.OOOOOO 


0.000000 


203 


2 




-0.036263 


-0.036557 


0.999997 


0. 535096 


203 


2 


9 


-0. 070963 


-0.071555 


0.999988 


0. 496935 


203 


2 


h 


9m 963905 


0.069005 


i.ooeooo 


O.OOOOOO 


203 


2 


J 


0. 034705 


0.034300 


0.999995 


0. 544439 


203 


2 


I 


0.002329 


0.002329 


1.000000 


2.00O0OO 


203 


2 


M 


-0. 044468 


-0. 04S014 


0.999990 


0. S26024 


203 


2 


n 


0.004985 


0.004512 


0.999996 


0. 710858 



P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: SlallaHty Measure: #1 - Cosine* #2 - Dice 
TV: Tera Weighting Scheaes: See Result Page R-1 

Correlation Coefficients: rj^ Is between the user's relevance judgwnt and the 
«ystea*s predicted reletance hascd on unresolved anaphors. rjr Is between 
the user's rtlevanct judgMnt and the systca's predicted relevance based 
ON resolved anaphors. 

Because the user's judgsniU were scaled fran low to high (1 » aost relevant, 
4 • m%t non-relevant) a stroM negative correlation shows agrecncnt 
betwten user's and system's relevance JudgMnts. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rjr> rjy). If this Z Is sutlstlcally 
significant as Indicated fey tie asterisks, then resolving awphors Imtovos 
the vttfli's predications of relevance. ^ 
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A Statistical Coaparlson of the Rtlatlonslilp Between 
Unresolved Anaphors and User's Relevance Judgaents with Resolved 
Anaphors and User's Relevance Judgaents: for Anaphoric Class 



Correlatio n Coefficients Significance Level 

^ "ju -^jr "^ur 2 l> > 





« 


• 


0. 0d9726 


0.068733 


0.999863 


0. 269908 


227 


• 




-0.006064 


0.032065 


0.992616 


-1.423458 


2»7 


• 


• 


-0.0O09S6 


-0.000956 


1.000000 


0.00)0000 


£97 


1 


h 


0.213822 


0.ai3874 




-0.850277 


307 


1 


J 


0. 168212 


0.174635 


0.999839 


-1.615627 


2«7 


« 


K 


0.039118 


0. 121884 


0.997082 


-1.338973 


2*7 


i 


n 


0.064855 


8.084285 


0.997783 


-2.307989 


2«7 


2 


a 


0.069602 


0.068528 


0.999717 


0. 202447 


207 


2 


b 


0.054739 


O. 054262 


0.999936 


0. 188977 


207 


2 


c 


0.070263 


0.069245 


0. 999769 


0.212202 


207 


2 


c 


0.021932 


0.048181 


0. 997133 


-1.551208 


207 


2 


• 


-0.000956 




1.000000 


O. 000000 


207 


2 


f 


-0.040281 


-0. 025129 


0.998944 


-1.475142 


207 


2 


S 


0.018805 


0.042610 


0. 997630 


-1.547063 


207 


2 


h 


0. 202720 


0.202787 


Am VWWW 


-1.2:0457 


207 


2 


J 


0. 193024 


0. 195876 


0.999971 


-1.702039 


207 


2 




0. 175987 


0. 176039 


1.000000 


-:. 043660 


207 


2 


M 


0. 121846 


0. 135433 


0. 999145 


-1.477537 


207 


2 


n 


0.098820 


0. 111597 


0. 999273 


-t. 504133 



NOTES: 

Q: Queries 100-199 iicre searched on INSPEC: 200-299 on PsychlNFO 
S: Slallarlty Neasiire: #1 • Cosine. #2 • Dice 
TN: Tera Melghtlng Sch«Ms: See RcstfU Page R-1 

Correlation Coefficients: rjy Is betoeen the user's relevance iudgwnt and the 
syste«*s predicted releVance tased on unresolved anaphors. rjr Is betMoen 
the user's relevance Judgamt and the systoa's predicted relevance hised 
on resolved amphors. 

Because the user's Judgatnts ware scaled froa low to high (1 • aost relevant. 
4 ■ aost non-relevant) a stroM negative correlation shows agreoNnt 
between user's and systea's relevance Judgaents. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
thin the first correlation (rjr> rj^). If this Z Is sUtlstlcally 
significant as Indicated ty tie asterisks, then resolving amphors laproves 
the i|yst«i*s predications of relevance. 
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Appendix 

A Stitlstlcal CoMparlsoR of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgaents with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 

APyERBS 

Correlation Coefficients Significance Level 



A 
V 


< 






"ur 


i P > 




1 




-0. 532393 


-0. 534533 


0.999965 


1.423616 


212 


I 


d 


-0. 630979 


-0. 640623 


0.997*44 


0.799694 


212 


• 


• 


261644 


-0. 331317 


0.964296 


1.427274 


212 


1 


h 


689775 


0.068470 


0.999987 


1.250639 


212 




J 


-6. 592539 


-0.604145 


0.997900 


1.067061 


212 




M 


-9. 663766 


-0. 677385 


0.996447 


1.505631 


212 


1 


n 


-6. 791465 


-0. 715708 


0.999080 


1.994759 <•♦♦♦ 


212 


2 


a 


-0- 553674 


-0. 556193 


0.999956 


1.510422 


2X2 


2 


b 


-6.536118 


-0. 537247 


0.999990 


1.407011 


212 


2 


c 


-6.560076 


-0. 562278 


0.999967 


1.525915 


212 


2 


d 


-6. 683127 


-0. 679519 


0.995743 


-0. 261418 


212 


2 


• 


-0.268411 


-0. 327933 


0.991660 


1.554373 


212 


2 


f 


-0. 681729 


-0. 676123 


0.995372 


-0. 387514 


212 


2 


9 


-0. 692679 


-0. 668354 


0.995545 


-0. 309736 


212 


2 




0.055266 


0.052574 


0. 999916 


1.030669 


212 




J 


-0. 685346 


-0. 684692 


0.998036 


-0. 070164 


212 


2 


1 


0. 004343 


0. 004002 


0.999999 


1.006618 


212 




m 


-0. 710906 


-0. 707554 


0. 996714 


-0. 236978 


212 


2 


'n 


-0. 731150 


-0. 726276 


0. 997792 


-0.5c:c^'32 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Slallarlty Neasurt: #1 • Cosine. #2 • Dice 
TM: Tera Weighting Schans: See Resttit Page R-1 

Correlation Coefficients: rju Is between the user's relevance judgMnt and the 
systca's predicted relevance tesed on unresolved anaphors. rjr Is between 
the user's rtlevanct judgMnt and the systoi's predicted relevance based 
on resolved anaphors. 

Because the user's judgoents were scaled froa low to high (1 • aost relevant, 
4 • aost non-relevant) a strona negative correlation shows agrcoacnt 
between user's and system's relevance JudgMnts. 

Significance Level: A potltlvo Z Indicates that the second correlation Is higher 
than the first correlation (rj,.> rj^). If this Z Is sutlstlcally 
significant as Indicated fey the asterisks » then resolving anaphors loproves 
the iyst«i*s predlcatlcns of relevance. 
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Appendix E-196 
Page 

A Statistical Co«par1soii of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgaents with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 

APyERBS 



Correlation Coefficients 



Significance Level 



Q 


s 






219 




219 




219 




219 




219 




2X9 




219 


2 


219 


2 


219 




219 


2 


219 


2 


219 


2 


219 


2 


219 


2 


219 


2 


219 


2 


219 




219 


2 



TW 

• -9 
d -0, 
• 

h -0. 
J 

«» -0. 
« -0. 

• -0. 

b -0. 

C -0. 

c -0. 

• -0. 

f -0. 
E -0. 
-0. 

J -®. 
1 -0. 
« -0. 
rt -0. 



33S41S 
431 19£ 
001101 
738016 
67165£ 
.54iaSS 
.562968 
.'253123 
.223322 
. 264359 
.395491 
001101 
380616 
404803 
769917 
594641 
616435 
504693 
520416 



-0.331750 
-0. 457892 
-0.001101 
-0. 728622 
-0. 679750 
-0. 557848 
-0. 578395 
-0. 254585 
-0. 224152 
-0. 265601 
-0. 423484 
-0.001101 
-0. 409756 
-0. 432203 
-0. 767515 
-0. 600761 
-0. 615349 
-0. 518620 
-0. 532003 



ur 

0. 999978 

0.993178 

1.000000 

0. 999181 

0.999325 

0.997718 

0.998266 

0. 999933 

0.999984 

0. 999944 

0. 992759 

1.000000 

0.991021 
0.992215 
0. 999397 

0.999318 
0. 999353 
0. 997799 
0. 998432 



P > .05 



0. 212136 
1.002152 

0.000000 

-1.252895 
1. 128190 
1.09426S 
1.217319 
*. 519852 
0. 606919 
0. 486586 
1. 004290 

0. 000000 

0. 935396 
0.953313 
-0 428214 

0. S0S325 
-0. 153117 
0. 955362 
0. 9S0606 



NOTES: 

Q: (hierles 100-199 were searched on IHSPEC: 200-299 on PsychlHFO 
S: Slallarlty Measure: «1 « COslne. #2 « Dice 
TH: Term Weighting Schcaes: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance Judgwnt and the 

systiM's predicted rr'-** " 

the user's relcvaiice , 
on resolved antphors. 

Because the user's judgMnU were scaled fro« low to high (1 • nost relevant, 
4 • Mst nonnrelevant) a stroM negative correlation shows agrccnent 
hetwten user's and systea's relevance judgwnts. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rjr>rjy). If this Z Is sUtlstlcally 

^J2?*"!^ fcy tie asterisks, then resolving anaphors 1i«>roves 
the vstiM's predications of relevance. w 



t: rju Is between the user's relevance Judgwnt and the 
relevance based on unresolved anaphors. nr Is between 
ee judgwnt and the systoa's predicted relevance based 
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Appendix E-197 



Page 

A Statistical Coaparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgaents with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 



APYERBS 



Correlation Coefficients 



Significance Level 



221 
221 
221 
221 

221 
221 
221 
221 
221 
221 
221 
221 
221 



221 



2 
2 
2 



2 
2 



TW 



• 9. 

d 9. 

• -9. 

^ -9. 
J 

m 9. 

n -9m 

• 9, 
b 9. 
c 9. 
d 9. 
» -9. 
f -9. 
a e. 

n -9. 

J 
1 

« -0. 

r. -9. 



944923 
073499 
099885 
934912 
972548 
912463 
982338 
93S5S9 
924968 
935314 
949988 
090685 
993799 
035177 
034210 
966140 
001531 
913498 
092854 



0. 044756 
0.068418 
-0.099865 
-9.934912 
-0. 978912 
9.906972 
-0. 985469 
0.935454 
0. 023727 
0.035149 
0. 039240 
-0. 000885 
-0.004296 
0. 033652 
-0. 034219 
-0. 067458 
-0. 001531 
-0. 014859 
-0. 093602 





7 


"^ur 




0.999961 


2. 071199 


0. 999671 


1.921262 


1.000000 


0.000000 


1.900900 


0.000000 


0.999855 


1.204255 


0. 999766 


1. 105415 


0.999962 


.:. 131846 


0. 999933 


0. 037527 


0., 999973 


0. 173684 


0. 999942 


0. 057249 


0 . 999922 


0. 524472 


1.000000 


0. 000000 


0. ?»959a6 


0. 347925 


0. 999936 


0. 505707 


1.000000 


0. 000000 


0.999979 


0. 754856 


1.000000 


0. 000000 


0. 999966 


0. 617133 


0. 999993 


0. 738241 



P > .05 



NOTES: 

Q: (^ies 100-199 were searched on INSPEC: 20O>299 on PsychlNFO 
S: Siailaritjf Measure: #1 « Cosine. #2 « Dice 
TH: Teni Weighting Schtws: See Result Page R-1 

is between the user's relevance JudgMnt and the 



Correlation Coefficients: 
systSM's predicted 
the user's relevance 
on resolved amphors. 



i: rju is between the user's relevance JudgMnt and the 
relevance based on unresolved anaphors. rw is between 
« judgncnt and the system's predicted relevance based 



Because the user's JudgwnU were scaled froa low to high (1 « nost relevant, 
4 • Mst non-relevant) a strong negative correlation shows agrecMnt 
between user's and systeii's relevence Judgaents. 

Significance Level: A positive Z indicates that the second correlation is higher 
than the first correlation (r4|.> rju). If this Z is sutistically 
significant as Indicated fey the asterisks, then resolving anaphors Inproves 
the lystM's predications of relevance. 
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Page 

A Statistical Ce«par1son of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgaents: for Anaphoric Class 



APyERBS 



Correlation Coefficients Significance Level 



Q 


s 


TM r. 


''ir 




z 








J*' 


J" 


ur 






1 






-9. 965393 


1.999999 


9. 999999 




4 


e 


407:37 


-9. 497137 


1.999990 


9.999999 










-9. 999884 


1.999999 


9. 999999 




♦ 


h 


-9. lazAZi 


-9. 162421 


1.999999 


9.999999 






J 


-9. 493334 


-9. 493334 


1.999999 


9. 999999 


S£S 


1 


m 


-9. 439S64 


-9. 439564 


1.999999 


9. 999999 


£52 




n 


-9. 134989 


-9. 134969 


1.999999 


9.999999 


222 




a 


-9. 978253 


-9.976253 


1.999999 


9. 999999 


222 






-9.983X33 


-9.963133 


1.999999 


9. 999999 


222 




c 


-9. 975564 


-9. 973564 


1.999999 


9. 999999 


222 


2 


c 


-9. 419848 


-9. 419846 


1.999999 


9. 999999 


222 


2 


e 


-9. 999884 


-9. 999884 


1.999999 


9. 999999 


222 




f 


-9. 338849 


-9. 338849 


1.999999 


9. 999999 


222 






-9.41958: 


-9.419581 


1.999999 


9. 999999 


222 






-9. 922398 


-9. 922398 


1.999999 


9. 999999 


222 






-9. 439879 


-9. 439679 


1.999999 


9. 999999 


222 




1 


-9. 992454 


-9. 992454 


1.999999 


9. 999999 


222 


2 


m 


-9. 4424^3 


-9. 442443 


I . 999999 


9. 999999 


222 


2 




-9. 256865 


-9. 256865 


1.999999 


9. 999999 



P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: SiMllarity Measure: #1 » Cosine. 12 « Dice 
TU: Teni Weighting SchcMs: See Result Page R-1 

Correlation Coefflcimts: rjy Is between the user's relevance judgnent and the 
system's predicted relevance based on unresolved anaphors. rj^ Is between 
the user's relevance J udj i in t and the systaa's predicted relevance based 
on resolved anaphors. 

Because the user's Judgwnts were scaled fro« low to high (1 ■ nost relevant, 
4 • iBSt non*re1evant) a streno negative correlation shows agrecnent 
between user's and system's relevance Judgaents. 

Significance Level: A positive Z indicates that the second correlation is higher 
than the first correlation (rjr> rju). If this Z is statistically 
significant as Indicated fey the asterisks, then resolving anaphors Inprovcs 
the nysteM's predicationi of relevance. 
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A Statistical Coiiparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgaents: for Anaphoric Class 



ADVERBS 



Correlation Coefficients Significance Level 



A 
W 


c 
5 


TW r. 




r 


z 












ur 




£S3 


1 




-9. 732594 


-0. 732594 


1. 000000 


V« WWW 


223 




d 


-0. £94873 


-0. 689378 


0. 999583 




SS3 


1 


• 


>0. 000648 


-0. 000648 


1.000000 


0. 000000 


223 


1 


h 


-0. 373036 


-0. 373061 


1.000000 


0. 175582 


233 


1 


J 


-0. 422509 


-0. 427477 


0. 999156 


0. 497421 


223 


1 


m 


>0. 695676 


-0.699367 


0. 998016 


0. 304605 


223 


1 


*> 


-0. 639829 


-0. 646247 


0. 997095 


9.408513 


223 


2 


• 


-0. 754136 


-0. 754136 


1.000000 


0. 000000 


223 


2 


b 


-0.718551 


-0. 718551 


1.000000 


0. 000000 


223 


2 


c 


-0. 752635 


-0. 752835 


1.000000 


0. 000000 


223 


2 


d 


-0. 732325 


-0. 729744 


0. 999907 


-0. 979208 


223 


2 


• 


-0. 000648 


-0. 000648 


1.000000 


0. 000000 


223 


2 


f 


-0. 709220 


-0. 707628 


0. 999940 


-0. 745860 


223 


2 


S 


-0, 731842 


-0. 729464 


0. 999921 


-0. 979905 


223 


2 


h 


-0.361258 


-0. 36:454 


0. 999987 


0. 231277 


£23 


2 


J 


~«. 389875 


-0. 391907 


0. 999764 


0. 379268 


223 


2 


1 


-0.315803 


316129 


0. 995994 


0. 357899 




2 


m 

• 


7£6fc5£ 


-0. 729550 


0. 999522 


9. 497603 


223 


2 


r. 


670259 


-0. 67425: 


0. 999297 


0. 530942 



P > .05 



NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Slnllarlty Measure: #1 » Cosine. 12 • Dice 
TW: Tern Weighting Schenes: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance Judgnient and the 
systen't predicted relevance based on unresolved anaphors. rw Is between 
the uMr's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

BKause the user's judgments were scaled from low to high (1 - most relevant, 
4 • mast non-relevant) a strong negative correlation shows agremiient 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation irwy rj^). If this Z Is statistically 
significant as Indicated by the asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



ADVERBS 



Correlation Coefficients Significance Lev#l 



Q 


S 


TW 






ur 


z 


D > .05 




4 


m 

Wk 












m mm f 


1 


w 












*£7 


1 


» 


-12. 000656 


-0. 000658 


: . 000000 


0* 000090 




££7 






0. 333290 


0. 3353 Z0 


0. 990081 


-0. 06A7S2 








J 


0. 053££3 


-0. 073£Q5 


0. 96739£ 




<###« 


££7 






-0.££7ei£ 


-0. 334139 


0. 967161 


1.90:058 






.1 




-Q. 0£:5£3 


-0. £-.4835 


0. 94a£8r 


S. iS6A5B 


<##»# 


££7 


£ 


a 


-0. :££705 


-0. :££705 


1 . 000000 


0. 000000 






£ 


b 


-0. i33S39 


-0. : 33699 


1 . 000000 


0* 000000 




££7 


£ 


c 


-0. l£&353 


-0. :££353 


1 . 000000 


0« 000000 




i:£7 


£ 


o 


-0. £S8£05 


-a. 350143 


0. 978771 


1.400076 




£27 


£ 


• 


-0. 000G5O 


-0. 000658 


1 . 000000 


0. 000000 




£57 


£ 


t 


-0. £.'3££78 


-0. 333:98 


0. 980£43 


193629 




££7 


£ 


c 


~0.£9010A 


-0. 351518 


0. 978304 


370179 




.•i£7 


£ 




0. 33:148 


2. 33£916 


0. 997930 


-0. 131937 




:£7 


£ 


J 


0. 0£3736 


-0. 0Q536£ 


0. 975891 


23 134 A 


<###>* 




£ 




0.0379 J £ 


0. 039d3£ 


0. 987456 


469406 




J:-:7 


£ 




-0. £14'21£ 


"0. :-:9770S 


0. 975£53 


1.720195 




.-.£7 


£ 




-0. 06655£ 


-0. 199129 


0. 956814 


2. 037332 


<##»# 



NOTES : 

Q: Querl€£ 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: SlMllarlty Measure: #1 « Cosine. 12 « Dice 
TM: Ten Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rj^ Is between the user's relevance judgment and the 
system s predicted relevance based on unresolved anaphors. rir Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 > most relevant, 
4 • most non-relevant) a strong negative correlation shows agremiient 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (r4,.> rj^). If this Z Is statistically 
significant as Indicated by the asterisks* then resolving anaphors Improves 



the system's predications of relevance. 
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Appendix E-201 

Page 

A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 

ADVERBS 

Correlation Coefficients Significance Level 
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s 


TU 




jr 


r 

ur 


2 






a 


-0. 373570 


-0. 373570 


1.000000 


0. 000000 


£30 






-a. 251 943 


-0. 269100 


0. 396351 


0. 902719 


230 




• 


~0, 000£00 


-0. 000600 


1 . 000000 


0. 000000 


A3* 
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0. 0767£9 


0. 076110 


0. 999996 


1.031122 




1 


J 


0. 058003 


0. 044673 


0. 998714 


1. 147541 






w 


"0. 102235 


-0. 146149 


0. 978426 


0. 970751 


.I3zr 




n 


-0. :S9797 


-0. 196683 


0. 990710 


0. 937975 


230 




a 


-0. 305620 


"0. 305620 


1.000000 


0. 000000 


230 
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-0. 234 S57 


-0. £34657 


1 . 000000 


0. 000000 


230 
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-0. 301274 


-0. 301274 


1.000000 


0. 000000 


232! 
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-0. 197532 


-0. 206214 


0. 999202 


0. 964329 
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2 
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-0. 000600 


-0. 000600 


1.000000 


0. 000000 


.130 


2 




-0. 1^9564 


-0. 153486 


0. 999852 
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230 
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C 


-•0. 196:59 


-0, 204202 


0. 999332 


0. 975792 


230 


2 




0. 064555 


0. 061481 


0. 999915 


1.028799 
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J 


0. £•74391 


0. 062550 


0. 998378 


0. 908082 


230 


2 


•5 


0. 073664 


0. 072332 


0. 999965 


0. 697923 


230 


2 




-0.075132 


~0. 092989 


0. 995770 


0. 849063 


230 


2 


r» 


-0. 137167 


-0. 148515 


0. 998314 


0. 859593 



P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Slnllarlty Measure: II - Cosine. #2 - Dice 
TW: Tem Weighting Scheaes: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance judgwnt and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgMnt and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 - most relevant, 
4 > most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z IndlcaUs that the second correlation Is higher 
than the first correlation (rjr> rju). If this Z Is statistically 
significant as Indicated by the asterisks, then resolving anaphors linroves 
the system's predications of relevance. 
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Pegt 

A Statistical Comparison of the Relationship Bttwccn 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 

ADVERBS 



Correlation Coefficients 



Significance Level 
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P > .05 
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NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFD 
S: Similarity Measure: #1 - Cosine. #2 « Dice 
TW: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 « most relevant, 
4 " most nen-re1evant| a strone negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (r«r> rj^). If this Z Is statistically 
significant at Indicated bjf m asterisks* then resolving anaphors Improves 
the system's predications of relevance. 
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Page 

A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 

ADVERBS 
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Significance Level 
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NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Slnllarlty Heasure: #1 « Cosine. 12 « Dice 
TM: Tern Weighting Schews: See Result Page R-1 

Correlation CoefflclcnU: rjy is between the user's relevance judgnent and the 
system's predicted romance based on unresolved anaphors. rjr Is between 
the user's relevance judgMnt and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgMflts were scaled fro« low to high (1 « Mst relevant, 
4 • nst non-relevant) a strong negative correlation shows agrecMnt 
between user's and systeai's relevance judgwnts. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (r«,.> rjy). If this Z Is sUtlstlcally 
significant as Indicated ly the asterisks, then resolving anaphors Iwroves 
the system's predications of relevance. 
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A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 
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Correlation Coefficients Significance Level 
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P > .05 



NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: II - Cosine. 12 - Dice 
TH: Term Weighting Schemes: See Result Page R-1 

Correlation Cbefficlento: rj^ Is between the user's relevance Judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the systoa's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 * most relevant, 
4 • most non-relevant) a strono negative correlation shows agremwnt 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation {rw> rj^). If this Z Is sUtlstlcally 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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Page 

A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 

DEFINITE ARTICLE 



Correlation Coefficients 



Significance Level 



S03 

£03 

£03 

293 

203 

£03 

£03 

£03 

£03 

£03 

203 

203 

£03 

£03 

£03 

£03 

£03 

£>25 

£03 



£ 

£ 
£ 
2 
£ 
£ 
£ 
£ 



TU 

d -0 

• -0. 
h 0. 

J 0. 

K -0. 

r« -0. 
a -0. 

0 0. 
c -0. 
ci -0. 

• -0. 
f -0. 
5 -0. 
t» 0. 

J 0. 

1 0. 
m -0. 
n 0. 



. 105343 
. I4XG£e 
.001022 

. : i70fi4 

072857 
097276 
012827 
042574 
005796 
037613 
070321 
001022 
036263 
070989 
069005 
034705 
002329 
044468 
004985 



-0. 106148 
-0. 161493 
-0. 001022 
0. 052089 
-0. 013138 
-0. 148119 
-0. 055664 
-0. 043710 
0. 005370 
-0. 038579 
-0. 127871 
-0.001022 
-0. 093124 
-0. 130450 
0. 048481 
-O. 052601 
0. 001232 
-0. 129594 
-0. 059381 



•"ur 


z 


P > 


0. 999980 






0. 991356 


0. 665910 




1.000000 


0. 000000 




0. 985430 


1.665079 




0. 989212 


2. 558135 


<»»»« 


0. 990305 


1.600324 




0. 994042 


1.720£15 




0. 999964 


0. 58£901 




0. 999995 


0. 574551 




0. 999974 


0. 585132 




0.981672 


1.316604 




1.000000 


0. 000000 




0. 981955 


1. 308198 




0. 98051 1 


1.3:9463 




0. 995322 


0. 9£64£7 




0. 978173 


1. 8£635l 




0. 999989 


1.031834 




0. 97600£ 


1.701714 




0. 984518 


1.597165 





.05 



NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: S1«11ar1ty Measure: II > Cosine. 12 « Dice 
TH: Terii Weighting Schenes: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance judgnent and the 
system's predicted remanct based on unresolved anaphors. rw Is between 
the user's relevance judgMnt and the systOM's predicted relevance based 
on resolved amphors. 

Because the user's judgMnts were scaled froM low to high (1 • most relevant, 
4 > Most non-relevant) a strong negative correlation shows agrecMcnt 
between user's and systea's relevance judgwnts. 

Significance Level: A positive Z Indicates that the second correlation is higher 
than the first correlation (r4,.> rjy). If this Z Is sUtlstlcally 
significant as Indicated by the asterisks, then resolving amphors linrovcs 
the systea's ^Icatlom ef rtievanee. . ^ 
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Page 

A Statistical Coaparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgaents with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



DEFINITE ARTICLE 



Correlation Coefficients 



Significance Level 
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P > .05 
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NOTES: 

Q: Queries lCO-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Slnllarlty Heasure: II « Cosine. 12 > Dice 
TU: Ter* Weighting SchoMs: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance judgment and the 
systen's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgmnt and the systea's predicted romance based 
on resolved anaphors. 

Because the user's judgnents were scaled fro« low to high (1 • nost relevant, 
4 • «ost non-relevant) a strong negative correlation shows agreoMnt 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (ri,.> rju). If this Z Is sUtistlcally 
significant as Indicated by the asterisks, then resolving anaphors Improves 
the ty%Um*% predications of relevance. 
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Page 

A Statistical COMparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 

DEFINITE ARTICLE 



Correlation Coefficients 



Significance Level 
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P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Slallarfty Measure: #1 • Cosine. 12 - Dice 
TH: Um Weighting SchoMs: See Result Page R-1 

Correlation Coefficients: rj^ Is between the user's relevance judgment and the 
systeM*i predicted relevance based on unresolved anaphors. rjr Is between 
the user's reltvance judgnent and the systen's predicted relevance based 
on refolvid anaphors. 

Because the ustr's judgMnti were scaled froia low to high (1 • nost relevant, 
4 ■ wst nonnrtlevint) a strong negative correlation shows agreanent 
between user's and system's relevance judgMnts. 

Signlfleanct Ltvel: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rjr> rjy). If this Z Is sutlstlcally 
ilgnlf leant as Indicated fey the asterisks, then resolving anaphors Improves 
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A Statistical Coaparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 



DEFINITE ARTICLE 



Correlation Coefficients 



Significance Level 
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NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: SlMllarlty Measure: II • Cosine. 12 - Dice 
TM: Tern Weighting Scheaes: See Result Page R>1 

Correlation Ceefflclents: rjy Is between the user's relevance judgment and the 
systea't predicted relefance based on unresolved anaphors. rjr Is between 
the user's relevance judgmnt and the system's predicted relevance based 
on resolved amphors. 

•ecause the user's judgments were scaled from low to high (1 • most relevant, 
4 • most fien-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level; A positive Z Indicates thit the second correlation Is higher 
than the first correlation (rjr> rju)* If this Z Is stetlstlcally 
f fgnlflcant as Indicated fey the asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 
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Correlation Coefficients Significance Level 
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-0. 062137 


0. 994236 


-0. 374685 



P > .05 



NOTES : 

Q: Queries 100-199 were starched on INSPEC: 200-299 on PsychlNFO 
S: SlMllarlty \^u%wrti #1 « Cosfne. 12 • Dice 
TN: Tern Ntlghtlng Schentf: Set Result Pagt R-1 

Corrtlatlon Cotfflclents: rji, Is bttwttn tht user's relevance Judgient and the 
systtn's prtdlcttd rtltl^anct bastd on unresolved anaphors. rw Is between 
tht user's rtltvanct judgMiit and tht systen's predicted relevance based 
on rtsolvtd anaphors. 

Itcaust tht ustr's judgMtnts wert scaltd fron low to high (1 • MOst relevant. 
4 « mst non-rtltvant) a strong ntgatlvt corrtlatlon shows agrconcnt 
bttwttn ustr's and systMi's rtltvanct judgMnts. 

Signlflcanct Ltvtl; A posftlvt Z Indfcatts that tht stcond corrtlatlon Is higher 
than tht first corrtlatlon {rwy rjy). If this Z Is sUtlstlcally 
tfgnff leant as IndlMttd tht asttrlsks, thtn rtsolving anaphors fmprovts 
tht lyttti's prtdlcatfons of rtltvanct* 
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Appendix E-210 



Page 

A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judg«ents: for Anaphoric Class 



DEFINITE ARTICLE 



Correlation Coefficients 



Significance Level 
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NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: SliHIarlty Measure: II • Cosine. 12 • Dice 
TM: Tern Weighting Schenes: See Result Page R-1 

Correlation Coefficients: rjy fs between the user's relevance Judgnent and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance Judg wn t and the system's predicted relevance based 
on resolved anaphors. 

lecause the user's judgments were scaled from low to high (1 « most relevant, 
4 • most non-rtlevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates thit the second correlation Is higher 
than the first correlation (rir> rju). If this Z Is sutlstlcally 
significant as Indicated by tm asterisks, then resolving anaphors Improves 
the iiystem's predications of relevance. 
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Page 

A Statistical Coiiparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 

DEFINITE ARTICLE 



Correlation Coefficients 



Significance Level 
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-0,754136 
-0. 718551 
-0. 752835 
-0, 654944 
-0. 000648 
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0. 930620 
1.000000 
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0. 932187 
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P > .05 



0. 000000 
-1.089483 
0. 000000 
1.684466 
0. 975742 
-1. 181094 
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0. 000000 
0. 000000 
0. 000000 
-1.038573 

0. 000000 

-0. 775813 
-1.032759 
1.747008 

0.613411 
1.707377 
-0. 985135 
-1. 137525 



NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Slidlarlty ffeasure; #1 • Cdslne. 12 • Dice 
TH: Terii Weighting ScheiMs: See Result Page R-1 

Correlation Ce«fff dents: rjy Is between the user's relevance judgnent and the 
systea's pridlcttd relefanct based on unresolved anaphors. rw Is between 
the user's rtlevinct judgMnt and the systen's predicted relevance based 
on resolved ampherf . 

lecauft tiM user's judgMnts were scaled fron low to high (1 • nost relevant, 
4 • «st non-relevant) a strono negative correlation shows agresMnt 
betMon user's and systeii's relevance judgnents. 

Significance Level: A positive Z Indicates^ tint the second correlation Is higher 
than the first correlation (r4,.> rju). If this Z Is sutlstlcally 

the asterisks, then resolving anaphors Improves 



significant ts Indicated fey 

tbe lystM's predlcstlom of relevance. 
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Appendix E>212 



Page 

A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgnents with Resolved 
Anaphors and User's Relevance Judgnents: for Anaphoric Class 



DEFINITE ARTICLE 



Correlation Coefficients 



Significance level 



0 


< 








7 


227 


X 




-e. i2as73 


-0. 128573 


1.000000 


0. 000000 


227 




d 


-0. 321199 


-0. 377591 


0. 925556 


0. 697176 


227 




• 


-0. 000658 


-0. 000658 


1.000000 


0. 000900 


227 


1 


h 


0. 333390 


0. 338081 


0. 990107 


-0. 158368 


227 


i 


J 


0. 0S9S23 


-0.001564 


0. 953721 


0. 895120 


227 




Hi 


-0. 227812 


-0. 2926S5 


0.911602 


0.714758 


227 


f 


r% 


-0. 061523 


-0. 123365 


0. 929369 


0. 739915 


227 


2 


m 


-0. 122705 


-0. 122705 


1.000000 


0. 000000 


£27 


2 


b 


-0. X 33699 


-0. 133699 


1.000000 


0. 00000(!> 


£27 


2 


c 


-0. 126353 


126353 


1.000000 


0. 000000 


227 


2 


c 


-0. 288205 


-0. 354686 


0.961845 


1. 1286315 


227 


2 


m 


-0. 000658 


-0. 000658 


1.000000 


0. 000000 


227 


2 


f 


-0. 282278 


-0. 336992 


0.974451 


1. 130055 


£27 


2 




-0. 29010* 


-0. 356649 


0. 962468 


1. 139455 


227 


2 




0. 331148 


0. 336556 


0. 997301 


-0. 346911 


227 


2 




0. 023736 


-0.025261 


0. 972956 


0. 94301 1 
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P > .05 



NOTES: 

Q: Queries 100-199 were searched on INSPCC; 200-299 on PsychlNFO 
S: Slirflarlty Measure: II • Cosine. 12 • Dice 
TH: Tern Melghting SchoMs: See Result Page R-1 

Correlation Coefficients: rju It between the user's relevance Judgment and the 
systM'f predicted relevance based on unresolved anaphors. rw Is betwten 
the user's relevance Judjuant and the systen's predicted relevance based 
on resolved aniphers. 

lecaufe the user's judgMonts were scaled fron low to high (1 • most relevant, 
4 • wit MAHrelevant) e strono negative correlation shows agrecMent 
betwoee user's and systea's relevance judgMnts. 

Signlffcafiee Level: A positive Z Indicates that the second correlation Is higher 
thie the first correlation (rjr> rju). If this Z Is sUtlstlcally 
figfilf leant at Indicated by the asterisks* then resolving anaphors liiproves 
the lyttMi't predlcatlenf of relevance. 
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Appendix E-213 
Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



DEFINITE ARTICLE 



Correlation Coefficients Significance Level 
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p > .05 



NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Heasure: #1 • Cosine. #2 • Dice 
TU: Term Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 • most relevant, 
4 • most non-relevant) a strono negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rjr> rju). If this Z Is statistically 
significant as Indicated by tne asterisks, then resolving anaphors Improves 
the SK*tem's predications of jrelevance. ^t^^ 
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Page 

A Statistical Comparison of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgments: for Anaphoric Class 



DEFINITE ARTICLE 



Correlation Coefficients Significance Level 
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Z P > .05 
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NOTES : 

Q: Queries 100-199 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Heasure: II - Cosine. 12 - Dice 
TW: Term Weighting Schemes: Sec Result Page R-1 

Correlation Coefficients: rju Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 « most relevant, 
4 ■ most non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (rjr> rju). If this Z Is statistically 
significant as Indicated by the asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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A Statistical Conparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgaents: for Anaphoric Class 



DEFINITE ARTICLE 



Correlation Coefficients Significance Level 
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NOTES: 

Q: Queries 100-ig9 were searched on INSPEC: 200-299 on PsychlNFO 
S: Similarity Measure: II « Cosine. #2 « Dice 
fU: Tern Weighting Schemes: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance judgment and the 
system's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance Judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 « most relevant, 
4 • aost non-relevant) a strong negative correlation shows agreement 
between user's and system's relevance judgments. 

Significance Level: A positive Z IndlcaUs that the second correlation Is higher 
than the first correlation (r4r> rjy). If this Z Is statistically 
signff leant as Indicated by tne asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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A Statistical Coiiparlson of the Relationship Between 
Unresolved Anaphors and User's Relevance Judgments with Resolved 
Anaphors and User's Relevance Judgaents: for Anaphoric Class 



DEFINITE ARTICLE 



Correlation Coefficients 



Signlflca.nce Level 
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NOTES: 

Q: Queries 100-199 were searched on INSPEC: 200>299 on PsychlNFO 
S: Slallarlty Measure: II » Cosine. 12 » Dice 
TW: Tern Weighting Schen^s: See Result Page R-1 

Correlation Coefficients: rjy Is between the user's relevance Judgment and the 
systen's predicted relevance based on unresolved anaphors. rjr Is between 
the user's relevance judgment and the system's predicted relevance based 
on resolved anaphors. 

Because the user's judgments were scaled from low to high (1 » most relevant, 
4 • most non-relevant) a strong negative correlation shows agreement 
between user's and syste«'s relevance judgments. 

Significance Level: A positive Z Indicates that the second correlation Is higher 
than the first correlation (r4,.> rjy). If this 2 Is sUtlstlcally 
significant as Indicated by tnt asterisks, then resolving anaphors Improves 
the system's predications of relevance. 
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